Friendly Reminder: For the most up-to-date information regarding this article, please use the following link: https://confluence.sportradar.ag/display/SK/Game+Stitching
Image stitching is the process of combining multiple photographic images with overlapping fields of view to produce a segmented panorama or high-resolution image.
We use two or three cameras depending on the distance from the court in which they were installed.
Stitching issues
Stitching issues usually occur due to a wrong calibration or camera movement in the arena and requires an intervention on our side remotely.
Typical stitching issues results with either a “duplication” or “blurriness” of the players during their movement, the lines on the court themselves or a stitching that seems good on the court but is wrong on the public view.
In an arena there can be either 2 or 3 cameras and the stitching issue occurs where the cameras overlap. If there are 2 cameras in the arena, the stitching issue will be between cameras 1 & 2. If there are 3 cameras, the stitching issue will be either between cameras 1 & 2, 2 & 3 or both.
1. Grab your Reference Images
When we notice a stitching issue or we must stitch an arena for the first time, the first thing we need to do is open the “Arena” tab within WebAdmin and open the details for that arena.
After we open the details for that particular arena, the next thing we need to do is open the “Cameras” section by click on the "Cameras" button located in the top right corner of WebAdmin.
After opening the “Cameras” section, you will see the following picture below and what you need to do is fetch new images and then set them as reference.
It’s good practice to check the new images before setting them as reference in the case that the new ones are not good enough to use for stitching. You can click on the image itself to switch back-and-forth between the new reference image and the current reference image. You should ask yourself the following questions when grabbing your reference images:
Is there enough overlap between the camera images? (If one or more cameras have moved, there may not be enough overlap to stitch cleanly)
Are there too many obstructions on the court to place points or lines? (i.e. Players blocking most of the court, commentator tables blocking large portions of the sidelines, etc.) If this is the case, then fetch again the image you need until there are no more or less obstructions.
Is the image clear enough to distinguish clear matching points on the court? (In the rare event that a camera goes out of focus, you may not be able to find enough clear features that match between the reference images).
After the previous step is done, you can begin the stitching process which consists of 3 steps. All of these steps you can find in the upper right corner menu of WebAdmin (Playing field, Matchings and Lines).
2. Playing field
In the "Playing field" section you’ll work on the newly fetched images from the previous step, and you’ll work with these images during all steps of the stitching process.
2.1 Open Playing Field
2.3 Select the court type corresponding to the playing field. You can accomplish this part of the process by looking at the playing field and notice the white lines on the floor.
Installation with volleyball court only (=> No drawing of the basketball court in the venue or volleyball court is not centered with the basketball court or size of the basketball court is not compatible with our court models).
- The court model FIBA-WITH-VOLLEYBALL is the one to select for all the volleyball venues in Europe and countries with a FIBA league. NCAA-WITH-VOLLEYBALL is used for volleyball venues in US.
2.4 Rotate the court if needed (This is accomplished by clicking on the rotation logo button on the right side of the “Court Type”)
Only court types that are FIBA, HC-ice-hockey and HC-ice-hockey-Olympic court can be rotated.
More explanation here: KeeCalib - Release note 2.1.1.
2.5. Place a minimum of 4 points on the court image at the correct position as shown on the court sketch on the right.
Be careful whether the points should be placed outside or inside of a particular line. At least 4 points are the vertices of a quadrilateral!
Please see example 2.3 and 2.4.
Installation with volleyball court only (=> No drawing of a basketball court in the venue or volleyball court not be centered with a basketball court or size of the basketball court and is not compatible with our court models).
- Select only the points of the volleyball court (Less than 4 points can be provided to the left and right cameras)
2.6. Once you are done, click on the “Save points” button which can be seen on the image below that is in the shape of a Floppy Disk icon.
2.7. Repeat the procedure (Steps 2.1 to 2.6) for all the images (either 2 or 3)
Important: Be aware not to select points that are not visible on the images (See examples 3 and 4 below)
Example N°2.1
In this arena example, we need to rotate the court (By clicking on the rotation logo at right of “Court type” area).
We can see a long line for points 28 and 30, and a small line for points 29 and 31.
Example N°2.2
In this arena example, we don’t need to rotate the court.
We can see a small line for points 26 and 32, and a long line for points 27 and 33.
Example N°2.3
Be careful with the position of points in Step 1. Invisible points should never be selected!
In the example below point N°26 is not visible because the player is hiding it, so there’s no need to select it.
Example N°2.4
The points behind the glass should never be placed. (The glass causes an issue on the location of the points)
As an explicit example of this would be: "KS-US-HARBORCENTER2" (The bottom left circle is cut). Points 22 and 23 should not be placed.
3. Matchings
The goal in this step of the calibration process is to find distinct points shared by both images.
The second step (Matchings) basically refers to “connecting” the areas where the cameras overlap. The reason we need to do this is because those are the locations where the stitching occurs. Depending on the number of cameras, the matchings are done between cameras 1 & 2 (A 2 cameras setup) and between camera 1 & 2 and 2 & 3 (A 3 cameras setup).
When the steps below need to be completed:
Upon first calibration.
If the cameras setup has been changed.
If one or more cameras have moved.
3.1. Open the “Matchings” section within WebAdmin
3.2. Select two reference frames (Make sure there is overlap)
3.3.a Manual Matchings
For this step you need to select the same point on two images where the camera overlaps with the best possible precision.
Generally, there is no rule that defines how many points should exactly be placed on the image in order to have the best stitching. Approximately somewhere between 6-12 points should be the best choice, but the number may vary based on the situation. Sometime less is more!
Pro Tip: Select points on the Top, Middle and Bottom part of the image for the matching tool to work best. A good distribution in the overlap area is the key to a good stitching.
Once you place a point, optionally add X, Y or Z axis for the upper and lower constraints! This helps the stitching program to compute the stitched image more accurately. In cases, where you do not achieve a good image stitching, adding the X, Y, and Z constraints may improve your stitching results. Please follow these two important rules when applying constraints:
Points placed on the ground (at court) level should have constraints of
-10 < Z < 10.Points placed above the ground level should have constraints above 0, depending on how high up a point is. The height is measured in centimeters (cm).
For example, if a point on the wall is 1.5 meters (150 cm) above the ground, the upper Z axis constraint should beZ < 150.
When you’re happy with the placement of a point and its constraints, press the space bar on your keyboard to confirm the point (it will turn green) and then begin placing your next point.
Once you are done, click on the "Save the points" button (By clicking on the Floppy disk icon) and then repeat the same steps for all the pairs of overlapping images (either 2 or 3).
Points that overlap should be as precise as possible (Same spot on both pictures). This means that the point on Camera 1 should match the point on Camera 2 as precise as possible in order for the matching tool to work best.
Pro Tip: Use the court lines, flooring, and logos to help you place the points at the same spots on both pictures as shown in the image below.
Example N°3.1 Stitching that seems good on the court but is wrong on the back wall.
Please ignore small shifts on ceiling (and light). In most cases we could improve this without decreasing the stitching on the court. If you only see a small shift on the top or the bottom of the image, please ignore that as well. The problem is that the object it too close to the camera.
Focus your attention on the court sidebar and the wall, along the stitching line.
To fix this, we need to add matching points on all of the overlap zones.
3.3.b Automatic Matchings on Reference Frames
The Manual Matching can be automatized for a overlap zone if no points have been clicked for this overlap zone.
For this just run a calibration and all overlap zones without matching points clicked will be computed.
These found points will be uploaded just after the end of the calibration, so please check this result in the debug calibration images. If it’s needed, remove any bad auto matching points found or add manual matching points.
If matching points are already clicked on the overlap zones, you can remove all this points and start the calibration process. In this case the automatic matchings will be activated.
3.3.c Automatic Matchings after a Production
Some matching points are automatically extracted while a production is running. Those points are saved at the end of the production and are available to be used at that step to help you to win time.
The matching points collected during a production are available until the end of the next production, where they will be replaced by the new collected matching points.
Click on the button
“Fetch auto matchings points” to download all the detected automatic matching points.
If these points are not available, you will see this red rectangle in the image below.
If an arena of this type of image are without detected points, please add some matching points manually (See 2.a Manual Matchings).The number of detected points can be between 100 and 500,
Save by clicking on the floppy disk logo icon.
4. Lines
Now the goal is to spot and mark long straight lines. Preferably one for each side of the court.
Refer to the example below to see how to place the points from one image to the other. As best as you can please be as precise as possible. You can delete the selected point by pressing the delete key on your keyboard. Once all the points of a line have been placed, press the space bar to confirm (like matching points, it will turn green) and start the next line.
Note: Lines shared by two images don't have to have points placed at the same location. They must simply be placed on the same line with some overlap.
4.1. Click on the “Lines” button
4.2. Click on all the images in order to select them all (If it is an arena with a 2 cameras setup, you have 2 pictures & if it is an arena with a 3 cameras setup, you have 3 pictures).
YOU MUST SELECT ALL OF THE IMAGES CORRESPONDING TO THE CAMERAS SETUP, NOT JUST 1 OR 2 OF THEM.
4.3. Click the “ Go - Launch multi view” button (This is the green button at the top of the page).
4.4.a Manual Line Input
4.4.a Place the points on the bottom of the court line (Which have to be numbered “0” on all 3 images) either from the inside or outside of the line (On the example to the right, the points are placed from the inside).
Important: Each line should have between 10 and 15 points for each image (This will depend on the line length).
Important: A line can only be visible on one or two images.
4.4.b Semi-Automatic Line Detector
To help you with this tedious task, you can use the semi-automatic line detector.
4.4.b.1 For this you need to select 2 points or more on the same line and same frame. Next you will click on the button shown on the image to the right, to start the detection.
Pro Tip: Sometimes it is possible to add more points on the line to help the detector to find the line.
4.4.b.2. The detection can take between 1 to 30 seconds, depending on the line length and the difficulty to find it.
At the start of the detection you will see this blue rectangle notification showing the line ID.
4.4.b.3. If the detection works, a green rectangle will inform you that the computation is completed.
4.4.b.4. If the detection fails, a red rectangle will provide you the line ID and the frame ID where the line detector failed.
Important Tips
- If the line detection isn't blocking after launching the line detector on one line, you can identify another line by clicking 2 points or more in all of the frames.
- The detection is running for the line ID on each frame, so it’s more efficient to identify the line by clicking 2 points or more in all frames where the line is visible.
- It’s possible to run the line detector on a line already found by the line detector, but the result will be the same.
4.5 Save
4.5 After you have placed all the points, set the constraints of “X”, “Y” or “Z” for all of the points. To do this just click on one of the points of a line and then adjust the “X=”, “Y=” or “Z= “value.
The points on the ground should have a constraint of “Z=0”. This will apply the constraint for every point you have placed on that particular line. Do the same for all the other lines you placed.
For lines that are placed higher than the ground, the same height rules apply as in Step 3.a. (100 cm = Z = 100)
4.6. Repeat the same steps as shown in steps 4.4 & 4.5 using the top court line (which should be numbered “1” on all 3 images) either all from the inside or outside of the line.
(Optional)
Repeat for any additional lines that are shared between images. This is usually not necessary for basketball courts but is necessary for hockey rinks.
Now, a line can contain many points, so an option to remove all points of a selected line has been added. Right click on the selected line and select the “Delete this line” option.
Example N°4.1
Image below is for a basketball FIBA arena.
Example N°4.2
Image shown below is of an Ice Hockey arena.
Important: Don't forget to save the changes after you complete every step of the process.
Example N°4.3 (semi-automatic input) <(Not yet Released!!!)>
Select 2 points at the ends of the line and click on the button to start the detection.
First a blue rectangle appears and then it becomes a green rectangle if the detection is successful. (See 4.a.2 to 4.a.5)
At this point, all new points are added on the detected line.
Don’t forget to add a line constraint as explained in step 4.5
5. Calibrate
The calibration part of the process is the final step for stitching. On this step we will process all data entries in the last 3 steps.
5.1. Click on the “Calibrate" button.
5.2 Click on the "Calibrate" button.
5.4a If you get a "Green rectangle" your calibration is done (but still needs to be checked).
If the calibration is done, you will then be able to see your result by clicking on “Verify stitched frame” button. Makes sure to check all reprojection errors by selecting the images in the "Images" drop down list and then click on the eye icon.
Each image will be opened in a new tab.
If the reprojection errors and stitched images are good, please go to the step 5.
Please use the "How to check if it’s a good Calibration" article to check if the calibration is good.
5.4.b If you get a "Red Rectangle", something was wrong during the calibration process
If something was wrong, the calibration has failed, and you will see this red rectangle with an error description. Some of the images can be opened by selecting an image on the "Images" list and then clicking on the "eye icon".
Please see the "How to Fix a bad calibration" article in order to solve this matter and restart again at step 5.2.
5.5. Click on the “Validate” button to push this calibration.
6. How to check if it’s a good Calibration
6.1. First, open the “Cam1_retroprojectionError.png” by selecting it in the "Images" list and then click on the eye icon to open it in a new tab.
6.2. On this image to the right we see very small arrows that have different meanings.
Red arrows: reprojection errors of points that were selected in the “Playing field” section.
Green arrows: reprojection errors of points that were selected in the “Matchings” section.
Blue arrows: reprojection errors of points that were selected in the “Lines” section.
Here, these arrows are so small that we can only see very small dots when viewing the image.
The size of this arrows is proportional to the reprojection error in the stitching image.
So, the smaller they are the better it is!
6.3. Next, open the same image but for the next camera (“Cam2_retroprojectionError.png”).
Here you will also notice on the image to the right that the arrows are so small that we only see dots.
So the smaller they are the better it is!
6.4. In the next step, we open and check the stitching result by clicking on the “Verify stitched image” button. This will open this image in a new tab.
6.5. For our final check we will need to check the stitched image. Using the image on the right we can see that the stitching is good and that there is no break in the lines (on the playing field and on the wall) and no shift between camera images.
If during the check of your calibration you see one or more big arrows (like in the Example N°6.1 of this part) or an issue on the stitched image, please fix it by referring to step 7.3 https://atriumsports.atlassian.net/wiki/spaces/SK/pages/156729345/Game+Stitching#7.-How-to-Fix-a-bad-calibration.
Example N°6.1:
1. First, we open the “Cam1_retroprojectionError.png” and we can see 2 big arrows.
We notice that the calibration isn't good.
We need to check all the images before we start to fix this calibration.
2. In this next image ( “Cam2_retroprojectionError.png”), we can see a problem, one of the “errors reprojection arrow” is too big.
3. So we need to fix this issue by clicking points on these images by selecting them in the “Playing field” section.
This is explained in step 7.3
4. When all problems are solved, we can open the stitched image and if the result is good we can save this calibration.
7. How to Fix a Bad Calibration
7.1. If the calibration is incorrect, you will see a red rectangle with a description of the error. Please read it carefully. In many cases the problem is explained and how to fix it as well.
7.2. Select and open all the “CamX_reprojectionError.png” Images.
If these images are not available, please use “CamX_reprojectionError_preoptimisation.png”.
7.3. On these images we will see some arrows in different colors:
Red arrows: reprojection errors of points that were selected in the “Playing field” section.
Green arrows: reprojection errors of points that were selected in the “Matchings” section.
Blue arrows: reprojection errors of points that were selected in the “Lines” section.
The size of the arrows is proportional to the size of the error!
7.3.a On this image we can see 4 reprojection errors for the Playing field.
Click the red arrows. A small arrow means that the error is small, and a long arrow means that the error is huge!
7.3.b On this image we can see 3 reprojection errors for the Matchings. Click the green arrows.
A small arrow means that the error is small and a long arrow means that the error is huge!
7.3.c On this image we can see 5 reprojection errors for the Lines.
Click the blue arrows. A small arrow means that the error is small, and a long arrow means that the error is huge!
7.4. Correct any issues in the necessary sections of the “Playing field”, “Matchings” or “Lines” and then try to re-calibrate.
8. Most Common/Recurrent Errors illustrated
Example N°8.1: Wrong Rotation of the FIBA Court Model
1. On this image (Image “Cam2_reprojectionError.png”), we can see that there are two huge red arrows. Here 2 points have probably be wrongly clicked on for ref_camcourt2 in the “Playing field” application.
2. The reason of these errors is a bad rotation of the court field and bad ID points that were clicked.
This error is explained in Part 1 (Playing-field), see examples N°2.1 and 2.2
3. The Fix: Click on the “rotate” button and click on points 28 and 30.
4. The here is the result after a new calibration. (Image “Cam2_reprojectionError.png” shown to the right).
Example N°8.2: Points are not Clicked at the Correct Position
1. On this image (Image “Cam1_reprojectionError.png”), we can see two huge red arrows. Here 2 points have probably be wrongly clicked on ref_camcourt1 in the “Playing field” application.
2. To fix this problem we need to do the following below:
Open Playing Field.
Select the reference frame (Here in this case would be ref_camcourt1) where we see the problem.
For more information, please read the part “1. Playing field”.
So, on this image we can see that 2 points (n°21 and n°20) are not located at a good place.
3. Fix the problem, save, and re-calibrate.
4. Re-check the “Cam1_reprojectionError.png”, “Cam2_reprojectionError.png” images and then stitched image.
Now all arrows are small, and the stitched image is good. We can now validate this calibration.
Example N°8.3: Stitching that seems to be good on the court but is wrong on the back wall
In this arena, we can see a distortion at the edge of the images (especially on the upper part of the image).
To fix it, we need to add some lines.
We can see that the result is better but not perfect.
So, after adding some corresponding points, we can see that the result is very good.
To fix this issue do the following steps below:
For each ref_camcourt select the court model FIBA-26meters and then save the points
Relaunch the calibration and check the reprojection errors images.
Example N°8.5: Client timeout error
If you see this error message, please increase the timeout time to 600 seconds (10 minutes).
If the problem don't disappear please, please contact us. The contact information is shown at the bottom of this article.
Example N°8.7: Adding points for the Matchings step that is showing bad precision
In this case, the point n°4 is visible in “cam 1” but NOT in “cam 2”.
The better solution is to select another position.
Example N°8.8: Points of the “Playing field” step are added on only one line
In this case the calibration will be very difficult, please don't click only co-linear points.
Here we can add point 10, 11, 26 and 32.
Example N°8.9: Incorrect points position in the line step
The ice hockey side is not showing a line in the corner, it is a curve, so don't add a line points on this part of the field.
Example N°8.10: Points behind the glass in the line step
The glass causes an issue on the location of the points, so the points behind the glass should never be placed.
Example N°8.11: Manual Matching point accuracy
To fix this, please put the point a the same place in two frames.
You can see that it is fixed on “Cam2_retroprojectionError.png”
Example N°8.12: Non-standard court dimensions
In this case, we noticed that in some situations the court can have non-standard dimensions. The error is indicating that the pixel error is too high.
The standard length is 28 meters. In order to stitch properly, you might want to select another court type to help the computation.
Pay attention to volleyball court lines for the 28 meters. The distance between the lines is like this, corresponding to the FIBA with Volleyball court type.
In this situation, we have a 26-meter length court type. You can observe this by looking at the same lines with court type: FIBA 26 meters with volley.
Known Issues
If you receive an error message, it’s likely that somewhere in the 3 steps a mistake was made and the program cannot compute the stitched image. If this occurs, go back to step 1 and proceed through the steps again, by following the information below:
In step 1
Check that you have the correct court type selected.
Check that you don’t have points that are close together and switched around. (In particular, the points that are placed at the top and bottom of the half court line.)
Check that you didn’t accidentally place a point from the wrong part of the court on the current image. (For example, each time you open an image, the system is ready for point 0 to be placed. However, that point should only be placed in Image 1 (left side of the court). It’s easy to accidentally place that point in Image 2 or 3.
In step 2
Check the constraints for all points are set correctly.
If you have a lot of points set too high on the far wall, try deleting some of them. It’s difficult to estimate the height of points at times and incorrectly positioned constrained points may cause issues.
Check that you don’t have stray points placed accidentally in random areas.
In step 3
Check constraints for all your lines.
Check that the points on a line are all placed on the correct side of a line (outside vs. inside).
If you have more than 2 lines, try deleting one or two excess ones. Again, sometimes less is more.
Sometimes it can happen that you perform all steps correctly and try to push them using the Calibration tool, but you still receive errors .
If you see one of those errors: 
then 
Connect to the server via SSH and launch the following commands:
|
check the ubuntu version of the server by typing in the command below.
|
if the ubuntu version is 18.10, then launch the following command:
|
If you see an error similar to the one below, you will need to use Matlab directly to stitch the arena. This is an unknown issue that is being worked on at the moment.
Black bars on the image - Escalate to Delivery
When black bars are showing on the calibrated image and a part of the court or audience stands are not visible, it means that the camera has moved.
This issue needs to be escalated to Delivery in order to move the entire camera setup.
Points of Contact are listed below
- Calibration team: Viewing Experience Squad
- In case of a back-end issue: Create a JIRA ticket in VIEW project section.
- Team members:
- Marco Calemme
- Robert Fischer
- Pascaline Parisot
- Front-end Team: Product Automation Squad
- In case of front-end issue: create a JIRA ticket in GAT project and select WEBA component.
Was this article helpful?
That’s Great!
Thank you for your feedback
Sorry! We couldn't be helpful
Thank you for your feedback
Feedback sent
We appreciate your effort and will try to fix the article