What is a Reader Regression Test and How to Troubleshoot an Unsuccessful Test

Friendly Reminder: For the most up-to-date information regarding this article, please use the following link: https://confluence.sportradar.ag/display/SK/Reader+regression+tests

What is the purpose of regression tests ?

Images used for regression tests are the images listed under "Reader's records" in the readers pages within Webadmin (see a detailed description of this page here: Scoreboard Readings: Arena reader analyze interface).

The tests are run each night on all arenas and they can be launched manually by clicking on the "trigger regression tests" button (if the number of image is important, you may need to increase the timeout value to 30 sec or so). The below image shows the interface within Webadmin showing the Reader's Records list.

How to understand a regression test

A test compares the values decoded by the OCR on an image to ground truth values (These are the correct values) set by a user.

The interface above indicates, for each image, if the output of the tests gave the exact same values as the one's set in the ground truth.

Using the image below the following information decodes what each pictogram means

• The pictogram is green when the decoded values match the values set in the ground truth.
• The pictogram is red when the values do not match which indicates that the process is not able to correctly decode the information in the image.
• The pictogram is orange when no ground truth has been set for the image.

When/How to add a test for an image?

• Every time you do a new calibration of the OCR for a reader, you should setup a regression test on the image you used for calibration
• Each time you notice that during a game that information is not correctly decoded from the scoreboards, you should add a test on an image from the corresponding timestamp.

Check the following like to understand how to capture an image to add to a reader's record: Scoreboard Readings: Arena reader analyze interface (link to relevant section)

How to set the ground truth

When clicking on the name of an image in the 'Reader's records" list,  a section will appear displaying details about the currently decoded information and the current value of the ground truth.

If there are no decoded values for a newly captured image, click on the "trigger regression tests" button

you can manually set the correct value for the ground truth or use the arrows to copy the decoded value (if they are correct!) to the ground truth.

Don't forget to click the "Save gt value" after filling the values.

What should you do when a test fails?

1.) If there was camera motion since the image was captured → discard the image from the tests (Please do not delete the image)

2.) If you expect the OCR to decode the image correctly → Fix the OCR calibration. Use the steps below to help resolve the matter.

    1.) Launch the scoreboard calibration tool (Click on the OCR button).
    2.) Calibrate the reader (Scoreboard Readings : Scoreboard calibration interface).
    3.) Push new reader configurations to the arena.
    4.) Launch a regression test by clicking on the "Trigger regression test" button.
    5.) Check the results of the test for all the images that were previously correctly decoded. 

        If one test that you expect to work fails, repeat these steps.
    6.) If game is live, trigger a reader "configreload" by running this command:

keecontrol.sh sbdX sp configreload 1 - replace X with the Reader you just modified.

• If despite re-calibrating the OCR, the decoded values continue to be wrong follow the steps below:

  1. Set the correct values to the ground truth.
  2. Discard the image from the tests
  3. Create a JIRA ticket in the VIEW project (https://jira.sportradar.ag/projects/VIEW/summary) mentioning arena name, reader name and image name (print_202303.....)