The depth map colors are constantly changing

OpenCV, Python and other image processing questions
Post Reply
megil
Posts: 12
Joined: Mon Dec 23, 2019 8:32 am

The depth map colors are constantly changing

Post by megil »

Hi everyone,

I'm following the stereopi tutorial. Colors are changing when creating a depth map. My hand is fixed in the picture that I gave the connection. But the color is becoming a little lighter. So even though the picture is fixed, the depth map color fluctuates constantly.

First ,second and third photo(my hand fixed, but the colors are changing.) ;

https://ibb.co/prLjmGy

There is an error in the software but I could not find. Is there someone who can help?

Thanks.

stereomaton
Posts: 165
Joined: Tue May 21, 2019 12:33 pm
Location: France

Re: The depth map colors are constantly changing

Post by stereomaton »

As the software tries to reconstruct map from two changing images (images in videos often fluctuate, especially with fluorescent lighting) the result can fluctuate too, I guess.
What is the order of magnitude of the depth estimation variance?
Stereophotographer and hacker
Despite my quite active participation in the forum, I am not in the StereoPi team
StereoPi Standard Edition + CM3Lite module + a few cameras

User avatar
Realizator
Site Admin
Posts: 549
Joined: Tue Apr 16, 2019 9:23 am
Contact:

Re: The depth map colors are constantly changing

Post by Realizator »

Hello Megil,
There are several reasons of colors variation.
1. Actually, colors you see is not actual depth map data. Let's look inside the colorization part of our code (lines 92-94 from stereopi-tutorial, script 6)

Code: Select all

92    disparity_grayscale = (disparity-local_min)*(65535.0/(local_max-local_min))
93    disparity_fixtype = cv2.convertScaleAbs(disparity_grayscale, alpha=(255.0/65535.0))
94    disparity_color = cv2.applyColorMap(disparity_fixtype, cv2.COLORMAP_JET)
Line 92 performs normalization. At this place we limit maximum disparity difference (local_max-local_min) by 65535, and minimum became 0 (zero).
Line 93 helps us to do appropriate numbers conversion to the range from 0 to 255. It is a range for the grayscale image.
And in the line 94 we colorize it. The "brighest" points (max disparities) became red, and "darkest" (black) became dark-blue.

That is, if you have a tiny fluctuation in your depth map, and one point with the smaller disparity appears, all colors "shifts" according to this change.
Actually, if you will do a 3D reconstruction, image will be stable, as while reconstruction you did not any normalization we use for colorizing.

There are several approaches to "fix" colors and avoid this small color shifting:
- Fix colorization range. In line 92 we use calculated values disparity-local_min, local_max and local_min. You can estimate them and use fixed values
- Do auto-adjustment on-the-go. We used this approach in our latest code version, stereopi-fisheye-robot, script 6. In this code we're remember the smallest and the biggest disparity values from the current session, and do color range adjustment.
Notice: actually you can use stereopi-fisheye-robot scripts for non-fisheye cameras, as I described here on our forum.

2. Actual depth map fluctuation.
Even with the fixed disparity and color range, your actual depth map can fluctuate in some image regions. As a rule, it is a regions with a poor light conditions, or with a very small color gradient. The only way here is to tune your depth map settings. As a rule, you need to decrease all parameters like SWS, prefilter cap and so on.
Here you can get a "trap" of a human perception. You see, with the higher filter values you get more nice-looking, smooth map. And most users prefer this way. But actually your map will be more stable (but not so nice-looking) with the smaller parameters.
Eugene a.k.a. Realizator

stereomaton
Posts: 165
Joined: Tue May 21, 2019 12:33 pm
Location: France

Re: The depth map colors are constantly changing

Post by stereomaton »

With the in-depth answer from Realizator, my question has less interest. Normalization part is what I supposed behind my question which is confirmed in the answer.
Nonetheless, it might be interesting for you to check actual values.
Stereophotographer and hacker
Despite my quite active participation in the forum, I am not in the StereoPi team
StereoPi Standard Edition + CM3Lite module + a few cameras

megil
Posts: 12
Joined: Mon Dec 23, 2019 8:32 am

Re: The depth map colors are constantly changing

Post by megil »

stereomaton wrote:
Wed Jan 08, 2020 5:17 pm
As the software tries to reconstruct map from two changing images (images in videos often fluctuate, especially with fluorescent lighting) the result can fluctuate too, I guess.
What is the order of magnitude of the depth estimation variance?
I didn't undersand what you said.

megil
Posts: 12
Joined: Mon Dec 23, 2019 8:32 am

Re: The depth map colors are constantly changing

Post by megil »

Realizator wrote:
Thu Jan 09, 2020 7:51 am
Hello Megil,
There are several reasons of colors variation.
1. Actually, colors you see is not actual depth map data. Let's look inside the colorization part of our code (lines 92-94 from stereopi-tutorial, script 6)

Code: Select all

92    disparity_grayscale = (disparity-local_min)*(65535.0/(local_max-local_min))
93    disparity_fixtype = cv2.convertScaleAbs(disparity_grayscale, alpha=(255.0/65535.0))
94    disparity_color = cv2.applyColorMap(disparity_fixtype, cv2.COLORMAP_JET)
Line 92 performs normalization. At this place we limit maximum disparity difference (local_max-local_min) by 65535, and minimum became 0 (zero).
Line 93 helps us to do appropriate numbers conversion to the range from 0 to 255. It is a range for the grayscale image.
And in the line 94 we colorize it. The "brighest" points (max disparities) became red, and "darkest" (black) became dark-blue.

That is, if you have a tiny fluctuation in your depth map, and one point with the smaller disparity appears, all colors "shifts" according to this change.
Actually, if you will do a 3D reconstruction, image will be stable, as while reconstruction you did not any normalization we use for colorizing.

There are several approaches to "fix" colors and avoid this small color shifting:
- Fix colorization range. In line 92 we use calculated values disparity-local_min, local_max and local_min. You can estimate them and use fixed values
- Do auto-adjustment on-the-go. We used this approach in our latest code version, stereopi-fisheye-robot, script 6. In this code we're remember the smallest and the biggest disparity values from the current session, and do color range adjustment.
Notice: actually you can use stereopi-fisheye-robot scripts for non-fisheye cameras, as I described here on our forum.

2. Actual depth map fluctuation.
Even with the fixed disparity and color range, your actual depth map can fluctuate in some image regions. As a rule, it is a regions with a poor light conditions, or with a very small color gradient. The only way here is to tune your depth map settings. As a rule, you need to decrease all parameters like SWS, prefilter cap and so on.
Here you can get a "trap" of a human perception. You see, with the higher filter values you get more nice-looking, smooth map. And most users prefer this way. But actually your map will be more stable (but not so nice-looking) with the smaller parameters.
Thank you for reply. I have guessed this.

stereomaton
Posts: 165
Joined: Tue May 21, 2019 12:33 pm
Location: France

Re: The depth map colors are constantly changing

Post by stereomaton »

megil wrote:
Thu Jan 09, 2020 12:50 pm
stereomaton wrote:
Wed Jan 08, 2020 5:17 pm
As the software tries to reconstruct map from two changing images (images in videos often fluctuate, especially with fluorescent lighting) the result can fluctuate too, I guess.
What is the order of magnitude of the depth estimation variance?
I didn't undersand what you said.
I can try to reformulate.

The depth map is an estimation which is constructed based on two images from two viewpoints. Since you use two videos, actuals values of the images are changing from frame to frame, thus the estimation is likely to change as well.
Fluorescent light (because it blinks) change image values a lot [as well as low light, because of image noise, by the way].
The question asked how much do the values (as depth and not as colors) change.
The idea behind this question was that the color scale might change depending on values (confirmed by Realizator) and in this case, perhaps that the actual variation of the estimated depths are not sufficiently big to worry.
Stereophotographer and hacker
Despite my quite active participation in the forum, I am not in the StereoPi team
StereoPi Standard Edition + CM3Lite module + a few cameras

Post Reply