Page 1 of 1

Colour Filtering Live Stream

Posted: Mon Dec 02, 2019 2:13 pm
by Alan_Hook
I have been working on a project for a while on other platforms (unity/oculus) that takes two live camera feeds in, processes them and then renders them into a headset. In the processing i want to remove the 'red' in the RGB, so that the video is yellow-green-blue spectrum. In the current version i use a LUT jpeg to shift the colour spectrum and remove all the red, (i then desaturate a little, drop the contrast etc)

I have been trying to rebuild the project in StereoPi. I have followed all of the tutorials for the DIY VR Headset (which are amazing and easy to follow for a beginner thanks).

I'm interested in what i could do in the YUV video to remove all of the red. I have tried a -cfx thinking that i could set a range with the two variables, and have done some testing with -awb off and -awbg custom settings after a reply on the Raspberry Pi main forum.

I am trying to remake this project

I'm working into the opt/StereoPi/scripts/video-source.sh

Code: Select all

	if [ "$rtsp_enabled" = "1" ] ; then
	    ./bin/raspivid -t 0 $DEC_STR $UP_DOWN -w $video_width -h $video_height -fps $video_fps --bitrate $video_bitrate --profile $video_profile $VMOD_STR $PREVIEW3D --intra $video_fps --inline --flush --exposure $exposure --sharpness $sharpness --awb $video_wb --contrast $contrast --digitalgain $digitalgain -o - | ./bin/rtsp-server > /dev/null 2>&1
	else
./bin/raspivid -3d tb -w 1080 -h 1920 -fs -t 0

Re: Colour Filtering Live Stream

Posted: Tue Dec 03, 2019 9:39 am
by Realizator
Hi Alan,
I've played with -cfx settings.
As I understand, these settings accepts a couple of numbers from 0 to 255, where first is U and second is V.
Image

So if we set -cfx 0:0 we get this image:
Image

And with -cfx 255:255 we have this one:

Image

After some analysis and looking at objects with different colors I can suggest, that -cfx settings actually choose just a single color, and use it to colorize Y-channel image (grayscale). Ideally we need a possibility to choose not a single color, but a range (like 0-255:0-128) to actually filter a color. I've asked this question at Raspberry Pi forum in your thread.
6by9 in this thread suggest you to play with AWB settings. It is a good way, but with manual AWB settings instead of auto you'll get troubles in a changing light conditions. I will try to find another way to do it. The worst (but working) path I see is to try to use simple C-based program, which will capture video, re-colorize it as you need, and show it on HDMI. But in our experiment you're following we used some hardware trick, when raspivid generate 1080x1920 image ignoring system HDMI settings (actually 1920x1080). So I'm in a process of finding alternative solution now.

Re: Colour Filtering Live Stream

Posted: Tue Dec 03, 2019 10:56 am
by Realizator
I've experimented with the AWB settings, suggested by 6by9. Looks like it works.

1. To turn off AWB and use manual blue and red correction you can use this command:
raspivid -awb off -awbg 1.0,1.0 -set
("-set" suffix prints current exposure, analogue gain, digital gain and AWB in console)
Notice: AWB numbers, printed in a console, are always the same, as we fixed them. I was unable to find the way to show these parameters without setting them. :-/

2. My playing with the numbers says, that the maximum values you can choose are 8.0.
If you try to set bigger value (like 9.0,9.0), you'll get error "Invalid command line option (-awbg)"

3. First digit is for RED, and second is for BLUE. My tests shows, that:
- Setting RED lower than 0.1 leads to bad exposure autosettings, and you got dark image
- Setting BLUE higher than 1.0 (like 2.0 or more) leads to blue color overfilling
- I guess GREEN component is always 1.0, so the best settings to start play with are:
raspivid -awb off -awbg 0.1,1.0
In this mode I can normally see all blue objects as blue, and green as green. All red objects looks like dark.

The only thing I've noticed is when you're changing light conditions, your green/blue balance can drift (more green as a rule).

Image1: Auto White Balance:
Image

Image 2: White Balance 1.0,1.0
Image

Image 3: Wite Balance 0.1,1.0
Image

I think you can use these settings as a starting point to find appropriate configuration.

Re: Colour Filtering Live Stream

Posted: Tue Dec 03, 2019 3:18 pm
by Alan_Hook
That's great, i think in some of the cfx settings i got a weird duel colour image, i'f i can replicate i'll post the image and the settings. It's a little irrelivant now, but an interesting effect. I think that this with some tweaks in saturation etc could work really well.

I am looking at making something like this, with the input on the left and output for screen on the right...

Image

So i think with -awbg i can make something similar with desaturation, contracts etc

Re: Colour Filtering Live Stream

Posted: Tue Dec 03, 2019 4:01 pm
by Realizator
I got your idea and agree, that it is possible to achieve similar results.
As 6by9 answered on RPi forum, camera sensor color channels sensitivity is different. So you can increase Blue from 1.0 to higher values to obtain some balance with the dominating Green. :-)

Re: Colour Filtering Live Stream

Posted: Wed Dec 04, 2019 6:17 pm
by stereomaton
I did not try it yet (planned though), but maybe you can also use PiCamera python module to get RGB version of the frames and do the image processing you need in GPU with OpenCV for example.
Displaying the result on HDMI should not be very hard and it would give you more flexibility in the process to apply.

Re: Colour Filtering Live Stream

Posted: Mon Dec 16, 2019 1:23 pm
by Realizator
stereomaton wrote:
Wed Dec 04, 2019 6:17 pm
I did not try it yet (planned though), but maybe you can also use PiCamera python module to get RGB version of the frames and do the image processing you need in GPU with OpenCV for example.
Displaying the result on HDMI should not be very hard and it would give you more flexibility in the process to apply.
Actually displaying on HDMI might be a problem, as Python lives in OS settings (1920x1080 screen), but we need 1080x1920 for this Waveshare device. May be we should fix raspivid to add color filtering option, like we did it with the -fs key (for displaying stereoscopic preview on non-3d screens).