DISFlow for disparity map

OpenCV, Python and other image processing questions
Post Reply
JeffWitz
Posts: 4
Joined: Thu Feb 25, 2021 9:35 pm

DISFlow for disparity map

Post by JeffWitz »

On a simple stereo system, the stereoBM juste compute a block matching and take the horizontal displacement field find for each BM.

The code is old and not optimal. There is so much more on OpenCV !
Starting with version 4.x, you can use DISFlow, which is really performant !

here the process to create the the associated flow

Code: Select all

import cv2
  DIS = cv2.DISOpticalFlow_create()
  DIS.setFinestScale(0)
  DIS.setPatchStride(1)
  DIS.setPatchSize(2)
  DIS.setGradientDescentIterations(10)
  DIS.setVariationalRefinementIterations(5)
  DIS.setVariationalRefinementDelta(1)
  DIS.setVariationalRefinementGamma(0)
  DIS.setVariationalRefinementAlpha(3)
The code perform a two stage registration technique. It use the classical pyramidal method i.e: it computes the flow on a lower scale and then initialize with the flow computed on the previous coarse scale.
The idea here is to add at each scale of the pyramid a step of variational refinement in order to have a smoother field.

The parameter choosen herein are parameters that I often use and works quite well.

Once you've set up the parameters, you can just launch the calc method

Code: Select all

flow=DIS.calc(img_left,img_right,init_flow.copy())
flow is a numpy array of size (w,h,2)
flow_init can help in case of too much differences between the pictures.

If you want to go realtime, you can choose to reduce DIS.setFinestScale(0)

it helps us a lot for fast and accurate flow

if you want to extract the disparity map
disp_map=flow[::,::,1]
feel free to ask questions !

User avatar
Realizator
Site Admin
Posts: 900
Joined: Tue Apr 16, 2019 9:23 am
Contact:

Re: DISFlow for disparity map

Post by Realizator »

Hi JeffWitz,
As for the DISFlow - thank you for highlighting this! I'll try to play with it!
Did you have a chance to play with it on Raspberry Pi? I'm curious about the performance of this approach. If you have a kind of tutorial (or can do a draft one) - this will be great!

As for the old code - looks like you are looking at our old OpenCV 3 code (https://github.com/realizator/stereopi-tutorial).
We have OpenCV 4-based code (our main repo now) here: https://github.com/realizator/stereopi-fisheye-robot
Eugene a.k.a. Realizator

JeffWitz
Posts: 4
Joined: Thu Feb 25, 2021 9:35 pm

Re: DISFlow for disparity map

Post by JeffWitz »

Did you have a chance to play with it on Raspberry Pi?
Yes of course ! For know I use it in order to register 2 images in mechanics (we call it Digital Image Correlation (DIC)). Once the flow is computed, we project it on projective tranformation in order to drive an experimental tensile test from a raspberry pi !
The raspberry pi process to build and control tensile test machine is not published yet, but we've done it using CRAPPy our laboratory software platform to control experiment.
You can try it on RPi with this program

I'm curious about the performance of this approach. If you have a kind of tutorial (or can do a draft one) - this will be great!
You can try with this code

Code: Select all

import cv2
# Set DISFLOW parameters
DIS = cv2.DISOpticalFlow_create()
DIS.setFinestScale(0)
DIS.setPatchStride(1)
DIS.setPatchSize(2)
DIS.setGradientDescentIterations(10)
DIS.setVariationalRefinementIterations(5)
DIS.setVariationalRefinementDelta(1)
DIS.setVariationalRefinementGamma(0)
DIS.setVariationalRefinementAlpha(3)

# Load images for test purpose
img_left = cv2.imread("My_left_camera_image",0)
img_right = cv2.imread("My_right_camera_image",0)
flow = DIS.calc(img_left,img_right,None)
# Extract the the good component of your rectified image 0 or 1
disparity_map=flow[::,::,1]
s for the old code - looks like you are looking at our old OpenCV 3 code (https://github.com/realizator/stereopi-tutorial).
We have OpenCV 4-based code (our main repo now) here: https://github.com/realizator/stereopi-fisheye-robot
What a misunderstanding ! I didn't talk about your code, but the StereoBM of OpenCV that is quite outdated in the image registration field ! Sorry for that !

It is one of the most optimized algo prior to the DeepLearning ones.
You can trade the accuracy for speed, tuning the parameter I give you.
If you want something very fast you can set :

Code: Select all

DIS.setFinestScale(1) # or even 2 !
DIS.setVariationalRefinementIterations(0)
if you want Variational improvement you have to increase

Code: Select all

DIS.setVariationalRefinementIterations(15) # ! it can increase the loop time drasticly
If you want a smooth field you can increase the alpha parameter in variational refinement

Code: Select all

DIS.setVariationalRefinementAlpha(20)
it will lead you to a gradient between the foreground and the background ... It should be balanced on each scene, but the first parameters should be good enough ...

If img_left is far from img_right, you can initialize with a flow that come from stereoBM (you can pad the vertical displacement with 0 or RBT ).

Is it clear enough ?

User avatar
Realizator
Site Admin
Posts: 900
Joined: Tue Apr 16, 2019 9:23 am
Contact:

Re: DISFlow for disparity map

Post by Realizator »

JeffWitz thank you! I'll try to play with this!
I need to figure out how we can get point cloud with this approach :)
Eugene a.k.a. Realizator

stereomaton
Posts: 215
Joined: Tue May 21, 2019 12:33 pm
Location: France

Re: DISFlow for disparity map

Post by stereomaton »

Shouldn't it be flow[...,0] to get the U component of the optical flow?
Is there a way to say that we are sure that the image is well aligned and that the flow should be computed only in X and not Y, or at least bias the cost function to prefer horizontal flow when possible?
Stereophotographer and hacker
Despite my quite active participation in the forum, I am not in the StereoPi team
StereoPi (v1) Standard Edition + CM3Lite module + a few cameras

User avatar
Realizator
Site Admin
Posts: 900
Joined: Tue Apr 16, 2019 9:23 am
Contact:

Re: DISFlow for disparity map

Post by Realizator »

Stereomaton, I think we can pass already rectified/calibrated image pair to get the best result. I hope to find some time to experiment with this approach.
Eugene a.k.a. Realizator

stereomaton
Posts: 215
Joined: Tue May 21, 2019 12:33 pm
Location: France

Re: DISFlow for disparity map

Post by stereomaton »

Yes, my second question was for this case: the optical flow searches for changes in 2D, but sometimes it might find vertical component if the grayscale image had not enough information to decide (uniform zone, occlusion...). That's why I asked if we can constrain it.
Stereophotographer and hacker
Despite my quite active participation in the forum, I am not in the StereoPi team
StereoPi (v1) Standard Edition + CM3Lite module + a few cameras

JeffWitz
Posts: 4
Joined: Thu Feb 25, 2021 9:35 pm

Re: DISFlow for disparity map

Post by JeffWitz »

Yes, my second question was for this case: the optical flow searches for changes in 2D, but sometimes it might find vertical component if the grayscale image had not enough information to decide (uniform zone, occlusion...). That's why I asked if we can constrain it.
No, you can't constrain it. I'm not sure it is the best thing to do. With the error due to the calibration process it is always interesting to be able to slightly move in the direction that should be 0 !

The Dense Inverse Search block matching part of this algo is the most robust block matching that I have ever use. It is possible to initialize the field with the previous one, that helps to have good values. We often test on one direction translation and there are no real issues.

I think it has no real interest to try to add the constrain of zero displacement in one direction.

Do you test it ? Does it give good results ?

stereomaton
Posts: 215
Joined: Tue May 21, 2019 12:33 pm
Location: France

Re: DISFlow for disparity map

Post by stereomaton »

I tested it in the past.

Here (a new test) with a photo from Perseverance robot on Mars:
Image

DISFlow gives this: (on the right part, I fused the original image for reference, and added the color wheel which says in which direction is the estimated displacement)
Image

The estimation looks quite good on the ground where there is a lot of texturing, but is completely bad in the sky with a lot of vertical displacement and wrong change of direction.

If we project on the X axis (which is getting only the u component of the flow), we get this: (again with the original fused for reference)
Image

Again, the ground looks quite good, but the sky is totally wrong and estimated very close.
I am not sure that a constrain to search only in the x axis would have helped much with this particular image, but I saw images where it would have very likely been better.

Anyway, it is better than stereoBM.
Stereophotographer and hacker
Despite my quite active participation in the forum, I am not in the StereoPi team
StereoPi (v1) Standard Edition + CM3Lite module + a few cameras

JeffWitz
Posts: 4
Joined: Thu Feb 25, 2021 9:35 pm

Re: DISFlow for disparity map

Post by JeffWitz »

@stereomaton

Really cool, the test you perform !

Are the images you post the real size one ?

I can work to drastically improve the quality. The code I made is only for fast assessment. I don't really use variational refinement and don't stop at the last scale.

So there is room for improvement.

If the picture are the good, I can try to have the best results and not focus on the speed !

User avatar
Realizator
Site Admin
Posts: 900
Joined: Tue Apr 16, 2019 9:23 am
Contact:

Re: DISFlow for disparity map

Post by Realizator »

Guys, looks like I need to do an experiment with the side-by-side comparison of StereoBM and DISFlow approaches. :-)
Eugene a.k.a. Realizator

Post Reply