Page 1 of 2

Depth map calibration/explanation

Posted: Wed Feb 19, 2020 8:59 am
by bpe
Hi!

I've been trying to get the StereoPi to work for a while now, and I would appreciate some help.
I'm using two Raspberry Pi v1.3 cameras and the StereoPi v0.9 board. Also, lately I've been using the updated code from the fisheye repository.

The images taken with the 2_chess_cycle script are very promising. Almost none of them were bad in terms of colors/lines, but those that were have been removed. When calling 4_calibration_fisheye, I ran into some errors. After some googling I managed to fix them by removing some of the image pairs that apparently were considered "bad pairs" - the corresponding error messages were "CALIB_CHECK_COND - Ill-conditioned matrix for input array X" - indicating the picture that was flawed, and later "abs_max < threshold in function stereoCalibrate", with no hint as to which image pair might cause the issue.
I managed to identify the problematic pair, and the script finished without errors. I then moved on to script number 5, and unfortunately, this is as far as I got. The output of the depth map is really underwhelming, and while I managed to improve it slightly by adjusting the bars in the interface, it's a looong way from being good. I added the initial depth map as well as the one I managed to generate as attachments. As you can see, the bottle is detected somewhat, but other than that, it's pretty bad. My question is, how to optimize these results.

I appreciate any input!

Re: Depth map calibration/explanation

Posted: Wed Feb 19, 2020 9:44 am
by Realizator
Hi Bpe,
According to your depth map on your first image, I see some "depth" inversion on your left edge of a table. Closest to the camera parts are blue, and far elements are red. This problem usually has 2 reasons:
1. Your left and right images are swapped. Please check you have the same camera capture settings for the calibration and for the capture.
2. You need to adjust MinDisp and NumOfDisp settings

If you can send me an archive with your images (ideally with the removed "bad" images) and a sample image, I will try to make an experiment and find other issues (if any). You can zip all files and put it somwhere in the cloud (Google drive etc.)

As for the bad images - I mentioned it in our last article.

By the way, your test image has a lot of white objects without textures (like wall and, partially, the table). These white zones are real problem for the depth map (not only for the StereoPi, but for a humans too :-) )

Re: Depth map calibration/explanation

Posted: Wed Feb 19, 2020 9:55 am
by bpe
Thanks for the quick response - I will look into your suggestions and will let you know!

As for the images, you can find them here:
https://drive.google.com/open?id=1SPP2Y ... 89cabJM-H1

Re: Depth map calibration/explanation

Posted: Wed Feb 19, 2020 10:00 am
by Realizator
Great, thank you! Got it!
Can you also please upload your "scenes" folder, and an image you've used for the depth map calibration?

Re: Depth map calibration/explanation

Posted: Wed Feb 19, 2020 10:04 am
by bpe
I updated the Drive.

As for your second suggestion, changing those values does something, but it's still far from what it should look like (in my opinion).
As for your first suggestion - I didn't make any chanes to the provided code, can you let me know where to look exactly?

Thanks!

Re: Depth map calibration/explanation

Posted: Wed Feb 19, 2020 10:08 am
by Realizator
Ok, I will download new files now.
As for the code - here is the line https://github.com/realizator/stereopi- ... une.py#L62
You can just swap imgLeft and imgRight and check, if it changes anything for you.

Re: Depth map calibration/explanation

Posted: Wed Feb 19, 2020 10:17 am
by Realizator
Bpe, what is the stereobase in your setup?

Re: Depth map calibration/explanation

Posted: Wed Feb 19, 2020 10:25 am
by bpe
Realizator wrote:
Wed Feb 19, 2020 10:17 am
Bpe, what is the stereobase in your setup?
Not sure what you mean, unfortunately :/

Re: Depth map calibration/explanation

Posted: Wed Feb 19, 2020 10:31 am
by Realizator
I mean the distance between your cameras.

Re: Depth map calibration/explanation

Posted: Wed Feb 19, 2020 11:12 am
by Realizator
Ok, looks like I found a problem source.
To say briefly, you've taken calibration chessboard images with one camera setup, but image for your depth map with another setup. Under "setup" I mean the positions of the cameras with respect to each other on your camera mounting plate.
That's why it is recommended to fix cameras as robust as possible, because if you move one of them, you need to recalibrate your setup.

What am I did:
1. I did a calibration based on your images set
2. I used your image with the bottle and a keybiard for DM tune, But after calibration I noticed, that key points (like a top of the bottle) is not equalized horisontally.
3. So instead of your bottle-keyboard image I used one of your scenes photo for Depth Map calibration (resized it from 1280x480 to 640x240 before).

Image

I used a "scenes" window on this screenshot as a ruler. You may compare the position of the bottle's cap on the left and right photos (left cap is higher), and the position of your hand on the chessboard photo (position on the right photo is higher).

So the idea is: fix cameras, so they can not move even slightly, after that do your calibration, and conduct your experiments. If you re-assemble your setup, or move one of your cameras, please do recalibration.
p.s. I decreased all DM settings to zero, except mindisp and numofdisp. THis helps to find appropriate settings faster. After that you can tune other parameters to get smoother DM.

Re: Depth map calibration/explanation

Posted: Wed Feb 19, 2020 11:40 am
by bpe
Well, this is quite the improvement, thanks so much! I will try to replicate it on my end and will let you know, probably by tomorrow afternoon.

In terms of camera movement, they are both stabilized with a black board to which they are screwed to, so I'd be shocked if they moved a little or even at all. Still, I will fix the screws, make extra sure everything is in place, and re-do the calibration. I'll let you know!

Re: Depth map calibration/explanation

Posted: Wed Feb 19, 2020 11:47 am
by Realizator
By the way, I mentioned your camera shift is vertical. V1 series camera has a small board with the sensor and optics, and sometimes this tiny board is not sticked to the green PCB. Try to move it slightly. As a rule, at the bottom of this sensor board it is 3M double-sided sticky tape with a protective layer. So you can remove this protective film and stick it to the PCB. Please check, if it is your issue.

Re: Depth map calibration/explanation

Posted: Wed Feb 19, 2020 1:37 pm
by bpe
So, I made sure the cameras are tightly in place, and they are. I re-screwed everything, just to be sure.
The cameras used to be unstuck, but that was fixed a while ago, so now there's no way they are moving.
I ran the calibration again with the exact same images, resized the image you used for the DM tuning and ran that, but with the exact same settings you have, it doesn't achieve the same results.
Any ideas?

Re: Depth map calibration/explanation

Posted: Wed Feb 19, 2020 2:40 pm
by Realizator
Am I right understand you, that you just try to repeat my experiment without taking any new photos? You’re using the same images set you provide me with your Google Drive? It’s a good idea to do this before taking the new images.
If you do it this way, have you try to adjust mindisp and numofdisp slightly? They are very sensitive.
Upd> I’m out of my lab now. I can export my depth map settings and send you this file tomorrow.

Re: Depth map calibration/explanation

Posted: Wed Feb 19, 2020 2:43 pm
by bpe
Yes, I did it with the exact same images I provided you with, so I thought by having the same DM settings I would get the same result. I tried a lot of tiny adjustments but got nowhere near your results.
I have not yet taken any new pictures.

Cool! Looking forward, thanks!

Re: Depth map calibration/explanation

Posted: Wed Feb 19, 2020 2:57 pm
by bpe
If it helps - I tried switching back imgLeft and imgRight, and I got this result. It's relatively close to what you got, only switched (red = far away, blue = close).

Re: Depth map calibration/explanation

Posted: Thu Feb 20, 2020 8:43 am
by Realizator
Hi Bpe,
My 3dmap_set.txt and dm-tune.jpg are in attached ZIP. I used downscaled scene_1280x480_7.png file for making this dm-tune.jpg

Re: Depth map calibration/explanation

Posted: Fri Feb 21, 2020 10:14 am
by bpe
Hi Eugene,

we noticed that our version of the code was slightly older than the one on your repository, so we updated it, reran the calibration and dm tuning with the images and your settings. Below is the result, which is very consistent with what you get.

Thanks a lot for the help!
dm.png
dm.png (83.56 KiB) Viewed 24521 times

Re: Depth map calibration/explanation

Posted: Fri Feb 21, 2020 12:10 pm
by Realizator
Ok, great!
Now, when your global settings are appropriate, you can do a fine tune of all other parameters.

Re: Depth map calibration/explanation

Posted: Sat Oct 24, 2020 5:50 am
by tuannm
hello Realizator,

Can you help me check the code "4_calibration_fisheye.py" for my images?
I do not know what happens when I received an error " (-3 internal error) calib_check_cond - ill-conditioned matrix for input array 35 in function 'cv::internal::calibrate extrinsic'". I used the latest code in your repository.
I did not use "cv2.fisheye.CALIB_CHECK_COND" in the code.

The desired output is rectified images for both the left and right images.
Please help me how to solve this issue.
Thank you.