Page 1 of 1

Stereo Equirectangular Projection Live Stream

Posted: Mon May 06, 2019 12:21 am
by wildag
This is just documenting some success for others.

I've managed to get my local gstreamer install to decode and modify the stereopi udp stream into a side-by-side stereo equirectangular projection.

I used the following article to build gst-plugins-vr with a fork of fixes for OSX.

https://medium.com/@yofujimo/how-to-mak ... fcab0bc020


Here is the command with a test viewer.

Code: Select all

gst-launch-1.0 -mv  udpsrc port=3000 buffer-size=300000 ! h264parse ! avdec_h264 ! glupload ! glcolorconvert ! hmdwarp ! gldownload ! videoconvert !  video/x-raw, width=1920, height=1080 ! autovideosink


And here is the command to forward the stream to another udp stream viewer

Code: Select all

gst-launch-1.0 -mv  udpsrc port=3000 buffer-size=300000 ! h264parse ! avdec_h264 ! glupload ! glcolorconvert ! hmdwarp ! gldownload ! videoconvert !  video/x-raw, width=1920, height=1080 ! x264enc !  rndbuffersize max=6500 ! udpsink host=DESTINATION_IP port=3000

Re: Stereo Equirectangular Projection Live Stream

Posted: Wed May 15, 2019 5:31 pm
by thaytan
wildag wrote: ā†‘
Mon May 06, 2019 12:21 am
This is just documenting some success for others.

I...

Here is the command with a test viewer.

Code: Select all

gst-launch-1.0 -mv  udpsrc port=3000 buffer-size=300000 ! h264parse ! avdec_h264 ! glupload ! glcolorconvert ! hmdwarp ! gldownload ! videoconvert !  video/x-raw, width=1920, height=1080 ! autovideosink
I'm surprised if you need the gldownload and friends to make this work on MacOS. This should work, without the GPU -> CPU -> GPU overhead:

gst-launch-1.0 -mv udpsrc port=3000 buffer-size=300000 ! h264parse ! avdec_h264 ! glupload ! glcolorconvert ! hmdwarp ! glimagesink

or, even better see if this works using the native H264 decoder:

gst-launch-1.0 -mv udpsrc port=3000 buffer-size=300000 ! h264parse ! vtdec ! glupload ! glcolorconvert ! hmdwarp ! glimagesink

Re: Stereo Equirectangular Projection Live Stream

Posted: Thu May 16, 2019 1:02 pm
by wildag
The purpose of the stream above is to push the image to an oculus or other VR headset. I believed that requires the gldownload step. Any other way to do it? Iā€™m not an expert

Re: Stereo Equirectangular Projection Live Stream

Posted: Sat May 18, 2019 10:29 am
by Realizator
thaytan wrote: ā†‘
Wed May 15, 2019 5:31 pm
I'm surprised if you need the gldownload and friends to make this work on MacOS. This should work, without the GPU -> CPU -> GPU overhead:
Thaytan, Wildag's task is to obtain equirectangular projection on the Oculus Go. Ideally we have only StereoPi (CM3 onboard) and Oculus Go (or Quest). So we have two options here:
1. Do this real-time image transformation onboard (CM3) and live stream already processed video.
2. Send video "as is" to Oculus, and do this transformation onboard.

His mentioned Mac implementation is a proof-of concept. I agree with you, that Mac has enough performance to do this without GPU. But both Raspberry Pi CM3 and Oculus are embedded systems with limited performance, so GPU acceleration is the only way to obtain FPS and latency needed.

Re: Stereo Equirectangular Projection Live Stream

Posted: Thu Mar 11, 2021 10:57 pm
by mango
I would love to replicate this on macOS (currently running Catalina). Did you do any tricks to install gst-plugins-vr?

Re: Stereo Equirectangular Projection Live Stream

Posted: Fri Mar 12, 2021 1:33 pm
by Realizator
Hmmm, interesting approach. Maybe our recordings from the 180 or 200-degree cameras can be viewed "as is".
I add this experiment to my to-do list.
As of now, all those transformations to be done on-the-go in our Oculus Quest application. We did a special shader for this conversion in a real-time.