Stereo Equirectangular Projection Live Stream

S.L.P. image questions, stereoscopic video livestream and recording, stereoscopic photo capture etc.
Post Reply
wildag
Posts: 19
Joined: Sat May 04, 2019 9:45 pm

Stereo Equirectangular Projection Live Stream

Post by wildag »

This is just documenting some success for others.

I've managed to get my local gstreamer install to decode and modify the stereopi udp stream into a side-by-side stereo equirectangular projection.

I used the following article to build gst-plugins-vr with a fork of fixes for OSX.

https://medium.com/@yofujimo/how-to-mak ... fcab0bc020


Here is the command with a test viewer.

Code: Select all

gst-launch-1.0 -mv  udpsrc port=3000 buffer-size=300000 ! h264parse ! avdec_h264 ! glupload ! glcolorconvert ! hmdwarp ! gldownload ! videoconvert !  video/x-raw, width=1920, height=1080 ! autovideosink


And here is the command to forward the stream to another udp stream viewer

Code: Select all

gst-launch-1.0 -mv  udpsrc port=3000 buffer-size=300000 ! h264parse ! avdec_h264 ! glupload ! glcolorconvert ! hmdwarp ! gldownload ! videoconvert !  video/x-raw, width=1920, height=1080 ! x264enc !  rndbuffersize max=6500 ! udpsink host=DESTINATION_IP port=3000

thaytan
Posts: 2
Joined: Wed May 15, 2019 4:55 pm

Re: Stereo Equirectangular Projection Live Stream

Post by thaytan »

wildag wrote:
Mon May 06, 2019 12:21 am
This is just documenting some success for others.

I...

Here is the command with a test viewer.

Code: Select all

gst-launch-1.0 -mv  udpsrc port=3000 buffer-size=300000 ! h264parse ! avdec_h264 ! glupload ! glcolorconvert ! hmdwarp ! gldownload ! videoconvert !  video/x-raw, width=1920, height=1080 ! autovideosink
I'm surprised if you need the gldownload and friends to make this work on MacOS. This should work, without the GPU -> CPU -> GPU overhead:

gst-launch-1.0 -mv udpsrc port=3000 buffer-size=300000 ! h264parse ! avdec_h264 ! glupload ! glcolorconvert ! hmdwarp ! glimagesink

or, even better see if this works using the native H264 decoder:

gst-launch-1.0 -mv udpsrc port=3000 buffer-size=300000 ! h264parse ! vtdec ! glupload ! glcolorconvert ! hmdwarp ! glimagesink

wildag
Posts: 19
Joined: Sat May 04, 2019 9:45 pm

Re: Stereo Equirectangular Projection Live Stream

Post by wildag »

The purpose of the stream above is to push the image to an oculus or other VR headset. I believed that requires the gldownload step. Any other way to do it? I’m not an expert

User avatar
Realizator
Site Admin
Posts: 900
Joined: Tue Apr 16, 2019 9:23 am
Contact:

Re: Stereo Equirectangular Projection Live Stream

Post by Realizator »

thaytan wrote:
Wed May 15, 2019 5:31 pm
I'm surprised if you need the gldownload and friends to make this work on MacOS. This should work, without the GPU -> CPU -> GPU overhead:
Thaytan, Wildag's task is to obtain equirectangular projection on the Oculus Go. Ideally we have only StereoPi (CM3 onboard) and Oculus Go (or Quest). So we have two options here:
1. Do this real-time image transformation onboard (CM3) and live stream already processed video.
2. Send video "as is" to Oculus, and do this transformation onboard.

His mentioned Mac implementation is a proof-of concept. I agree with you, that Mac has enough performance to do this without GPU. But both Raspberry Pi CM3 and Oculus are embedded systems with limited performance, so GPU acceleration is the only way to obtain FPS and latency needed.
Eugene a.k.a. Realizator

mango
Posts: 5
Joined: Tue Feb 23, 2021 5:12 am

Re: Stereo Equirectangular Projection Live Stream

Post by mango »

I would love to replicate this on macOS (currently running Catalina). Did you do any tricks to install gst-plugins-vr?

User avatar
Realizator
Site Admin
Posts: 900
Joined: Tue Apr 16, 2019 9:23 am
Contact:

Re: Stereo Equirectangular Projection Live Stream

Post by Realizator »

Hmmm, interesting approach. Maybe our recordings from the 180 or 200-degree cameras can be viewed "as is".
I add this experiment to my to-do list.
As of now, all those transformations to be done on-the-go in our Oculus Quest application. We did a special shader for this conversion in a real-time.
Eugene a.k.a. Realizator

Post Reply