Depth map calibration/explanation

OpenCV, Python and other image processing questions
Post Reply
bpe
Posts: 9
Joined: Wed Feb 19, 2020 8:41 am

Depth map calibration/explanation

Post by bpe » Wed Feb 19, 2020 8:59 am

Hi!

I've been trying to get the StereoPi to work for a while now, and I would appreciate some help.
I'm using two Raspberry Pi v1.3 cameras and the StereoPi v0.9 board. Also, lately I've been using the updated code from the fisheye repository.

The images taken with the 2_chess_cycle script are very promising. Almost none of them were bad in terms of colors/lines, but those that were have been removed. When calling 4_calibration_fisheye, I ran into some errors. After some googling I managed to fix them by removing some of the image pairs that apparently were considered "bad pairs" - the corresponding error messages were "CALIB_CHECK_COND - Ill-conditioned matrix for input array X" - indicating the picture that was flawed, and later "abs_max < threshold in function stereoCalibrate", with no hint as to which image pair might cause the issue.
I managed to identify the problematic pair, and the script finished without errors. I then moved on to script number 5, and unfortunately, this is as far as I got. The output of the depth map is really underwhelming, and while I managed to improve it slightly by adjusting the bars in the interface, it's a looong way from being good. I added the initial depth map as well as the one I managed to generate as attachments. As you can see, the bottle is detected somewhat, but other than that, it's pretty bad. My question is, how to optimize these results.

I appreciate any input!
Attachments
dm_new.JPG
dm_new.JPG (54.86 KiB) Viewed 1639 times
dm_initial.JPG
dm_initial.JPG (67.34 KiB) Viewed 1639 times

User avatar
Realizator
Site Admin
Posts: 450
Joined: Tue Apr 16, 2019 9:23 am
Contact:

Re: Depth map calibration/explanation

Post by Realizator » Wed Feb 19, 2020 9:44 am

Hi Bpe,
According to your depth map on your first image, I see some "depth" inversion on your left edge of a table. Closest to the camera parts are blue, and far elements are red. This problem usually has 2 reasons:
1. Your left and right images are swapped. Please check you have the same camera capture settings for the calibration and for the capture.
2. You need to adjust MinDisp and NumOfDisp settings

If you can send me an archive with your images (ideally with the removed "bad" images) and a sample image, I will try to make an experiment and find other issues (if any). You can zip all files and put it somwhere in the cloud (Google drive etc.)

As for the bad images - I mentioned it in our last article.

By the way, your test image has a lot of white objects without textures (like wall and, partially, the table). These white zones are real problem for the depth map (not only for the StereoPi, but for a humans too :-) )
Eugene a.k.a. Realizator

bpe
Posts: 9
Joined: Wed Feb 19, 2020 8:41 am

Re: Depth map calibration/explanation

Post by bpe » Wed Feb 19, 2020 9:55 am

Thanks for the quick response - I will look into your suggestions and will let you know!

As for the images, you can find them here:
https://drive.google.com/open?id=1SPP2Y ... 89cabJM-H1

User avatar
Realizator
Site Admin
Posts: 450
Joined: Tue Apr 16, 2019 9:23 am
Contact:

Re: Depth map calibration/explanation

Post by Realizator » Wed Feb 19, 2020 10:00 am

Great, thank you! Got it!
Can you also please upload your "scenes" folder, and an image you've used for the depth map calibration?
Eugene a.k.a. Realizator

bpe
Posts: 9
Joined: Wed Feb 19, 2020 8:41 am

Re: Depth map calibration/explanation

Post by bpe » Wed Feb 19, 2020 10:04 am

I updated the Drive.

As for your second suggestion, changing those values does something, but it's still far from what it should look like (in my opinion).
As for your first suggestion - I didn't make any chanes to the provided code, can you let me know where to look exactly?

Thanks!

User avatar
Realizator
Site Admin
Posts: 450
Joined: Tue Apr 16, 2019 9:23 am
Contact:

Re: Depth map calibration/explanation

Post by Realizator » Wed Feb 19, 2020 10:08 am

Ok, I will download new files now.
As for the code - here is the line https://github.com/realizator/stereopi- ... une.py#L62
You can just swap imgLeft and imgRight and check, if it changes anything for you.
Eugene a.k.a. Realizator

User avatar
Realizator
Site Admin
Posts: 450
Joined: Tue Apr 16, 2019 9:23 am
Contact:

Re: Depth map calibration/explanation

Post by Realizator » Wed Feb 19, 2020 10:17 am

Bpe, what is the stereobase in your setup?
Eugene a.k.a. Realizator

bpe
Posts: 9
Joined: Wed Feb 19, 2020 8:41 am

Re: Depth map calibration/explanation

Post by bpe » Wed Feb 19, 2020 10:25 am

Realizator wrote:
Wed Feb 19, 2020 10:17 am
Bpe, what is the stereobase in your setup?
Not sure what you mean, unfortunately :/

User avatar
Realizator
Site Admin
Posts: 450
Joined: Tue Apr 16, 2019 9:23 am
Contact:

Re: Depth map calibration/explanation

Post by Realizator » Wed Feb 19, 2020 10:31 am

I mean the distance between your cameras.
Eugene a.k.a. Realizator

User avatar
Realizator
Site Admin
Posts: 450
Joined: Tue Apr 16, 2019 9:23 am
Contact:

Re: Depth map calibration/explanation

Post by Realizator » Wed Feb 19, 2020 11:12 am

Ok, looks like I found a problem source.
To say briefly, you've taken calibration chessboard images with one camera setup, but image for your depth map with another setup. Under "setup" I mean the positions of the cameras with respect to each other on your camera mounting plate.
That's why it is recommended to fix cameras as robust as possible, because if you move one of them, you need to recalibrate your setup.

What am I did:
1. I did a calibration based on your images set
2. I used your image with the bottle and a keybiard for DM tune, But after calibration I noticed, that key points (like a top of the bottle) is not equalized horisontally.
3. So instead of your bottle-keyboard image I used one of your scenes photo for Depth Map calibration (resized it from 1280x480 to 640x240 before).

Image

I used a "scenes" window on this screenshot as a ruler. You may compare the position of the bottle's cap on the left and right photos (left cap is higher), and the position of your hand on the chessboard photo (position on the right photo is higher).

So the idea is: fix cameras, so they can not move even slightly, after that do your calibration, and conduct your experiments. If you re-assemble your setup, or move one of your cameras, please do recalibration.
p.s. I decreased all DM settings to zero, except mindisp and numofdisp. THis helps to find appropriate settings faster. After that you can tune other parameters to get smoother DM.
Eugene a.k.a. Realizator

bpe
Posts: 9
Joined: Wed Feb 19, 2020 8:41 am

Re: Depth map calibration/explanation

Post by bpe » Wed Feb 19, 2020 11:40 am

Well, this is quite the improvement, thanks so much! I will try to replicate it on my end and will let you know, probably by tomorrow afternoon.

In terms of camera movement, they are both stabilized with a black board to which they are screwed to, so I'd be shocked if they moved a little or even at all. Still, I will fix the screws, make extra sure everything is in place, and re-do the calibration. I'll let you know!

User avatar
Realizator
Site Admin
Posts: 450
Joined: Tue Apr 16, 2019 9:23 am
Contact:

Re: Depth map calibration/explanation

Post by Realizator » Wed Feb 19, 2020 11:47 am

By the way, I mentioned your camera shift is vertical. V1 series camera has a small board with the sensor and optics, and sometimes this tiny board is not sticked to the green PCB. Try to move it slightly. As a rule, at the bottom of this sensor board it is 3M double-sided sticky tape with a protective layer. So you can remove this protective film and stick it to the PCB. Please check, if it is your issue.
Eugene a.k.a. Realizator

bpe
Posts: 9
Joined: Wed Feb 19, 2020 8:41 am

Re: Depth map calibration/explanation

Post by bpe » Wed Feb 19, 2020 1:37 pm

So, I made sure the cameras are tightly in place, and they are. I re-screwed everything, just to be sure.
The cameras used to be unstuck, but that was fixed a while ago, so now there's no way they are moving.
I ran the calibration again with the exact same images, resized the image you used for the DM tuning and ran that, but with the exact same settings you have, it doesn't achieve the same results.
Any ideas?
Attachments
dm_new.JPG
dm_new.JPG (181.24 KiB) Viewed 1616 times

User avatar
Realizator
Site Admin
Posts: 450
Joined: Tue Apr 16, 2019 9:23 am
Contact:

Re: Depth map calibration/explanation

Post by Realizator » Wed Feb 19, 2020 2:40 pm

Am I right understand you, that you just try to repeat my experiment without taking any new photos? You’re using the same images set you provide me with your Google Drive? It’s a good idea to do this before taking the new images.
If you do it this way, have you try to adjust mindisp and numofdisp slightly? They are very sensitive.
Upd> I’m out of my lab now. I can export my depth map settings and send you this file tomorrow.
Eugene a.k.a. Realizator

bpe
Posts: 9
Joined: Wed Feb 19, 2020 8:41 am

Re: Depth map calibration/explanation

Post by bpe » Wed Feb 19, 2020 2:43 pm

Yes, I did it with the exact same images I provided you with, so I thought by having the same DM settings I would get the same result. I tried a lot of tiny adjustments but got nowhere near your results.
I have not yet taken any new pictures.

Cool! Looking forward, thanks!

bpe
Posts: 9
Joined: Wed Feb 19, 2020 8:41 am

Re: Depth map calibration/explanation

Post by bpe » Wed Feb 19, 2020 2:57 pm

If it helps - I tried switching back imgLeft and imgRight, and I got this result. It's relatively close to what you got, only switched (red = far away, blue = close).
Attachments
dm_new.JPG
dm_new.JPG (71.83 KiB) Viewed 1607 times

User avatar
Realizator
Site Admin
Posts: 450
Joined: Tue Apr 16, 2019 9:23 am
Contact:

Re: Depth map calibration/explanation

Post by Realizator » Thu Feb 20, 2020 8:43 am

Hi Bpe,
My 3dmap_set.txt and dm-tune.jpg are in attached ZIP. I used downscaled scene_1280x480_7.png file for making this dm-tune.jpg
Attachments
3dmap-set-dm-tune-jpg.zip
(46.64 KiB) Downloaded 29 times
Eugene a.k.a. Realizator

bpe
Posts: 9
Joined: Wed Feb 19, 2020 8:41 am

Re: Depth map calibration/explanation

Post by bpe » Fri Feb 21, 2020 10:14 am

Hi Eugene,

we noticed that our version of the code was slightly older than the one on your repository, so we updated it, reran the calibration and dm tuning with the images and your settings. Below is the result, which is very consistent with what you get.

Thanks a lot for the help!
dm.png
dm.png (83.56 KiB) Viewed 1583 times

User avatar
Realizator
Site Admin
Posts: 450
Joined: Tue Apr 16, 2019 9:23 am
Contact:

Re: Depth map calibration/explanation

Post by Realizator » Fri Feb 21, 2020 12:10 pm

Ok, great!
Now, when your global settings are appropriate, you can do a fine tune of all other parameters.
Eugene a.k.a. Realizator

Post Reply