Stereo Equirectangular Projection Live Stream

[wildag]

This is just documenting some success for others.

I've managed to get my local gstreamer install to decode and modify the stereopi udp stream into a side-by-side stereo equirectangular projection.

I used the following article to build gst-plugins-vr with a fork of fixes for OSX.

https://medium.com/@yofujimo/how-to-mak ... fcab0bc020


Here is the command with a test viewer.

Code: Select all

gst-launch-1.0 -mv  udpsrc port=3000 buffer-size=300000 ! h264parse ! avdec_h264 ! glupload ! glcolorconvert ! hmdwarp ! gldownload ! videoconvert !  video/x-raw, width=1920, height=1080 ! autovideosink

And here is the command to forward the stream to another udp stream viewer

Code: Select all

gst-launch-1.0 -mv  udpsrc port=3000 buffer-size=300000 ! h264parse ! avdec_h264 ! glupload ! glcolorconvert ! hmdwarp ! gldownload ! videoconvert !  video/x-raw, width=1920, height=1080 ! x264enc !  rndbuffersize max=6500 ! udpsink host=DESTINATION_IP port=3000

[thaytan]

This is just documenting some success for others.

I...

Here is the command with a test viewer.

Code: Select all

gst-launch-1.0 -mv  udpsrc port=3000 buffer-size=300000 ! h264parse ! avdec_h264 ! glupload ! glcolorconvert ! hmdwarp ! gldownload ! videoconvert !  video/x-raw, width=1920, height=1080 ! autovideosink

I'm surprised if you need the gldownload and friends to make this work on MacOS. This should work, without the GPU -> CPU -> GPU overhead:

gst-launch-1.0 -mv udpsrc port=3000 buffer-size=300000 ! h264parse ! avdec_h264 ! glupload ! glcolorconvert ! hmdwarp ! glimagesink

or, even better see if this works using the native H264 decoder:

gst-launch-1.0 -mv udpsrc port=3000 buffer-size=300000 ! h264parse ! vtdec ! glupload ! glcolorconvert ! hmdwarp ! glimagesink

[wildag]

The purpose of the stream above is to push the image to an oculus or other VR headset. I believed that requires the gldownload step. Any other way to do it? I’m not an expert

[Realizator]

I'm surprised if you need the gldownload and friends to make this work on MacOS. This should work, without the GPU -> CPU -> GPU overhead:

Thaytan, Wildag's task is to obtain equirectangular projection on the Oculus Go. Ideally we have only StereoPi (CM3 onboard) and Oculus Go (or Quest). So we have two options here:
1. Do this real-time image transformation onboard (CM3) and live stream already processed video.
2. Send video "as is" to Oculus, and do this transformation onboard.

His mentioned Mac implementation is a proof-of concept. I agree with you, that Mac has enough performance to do this without GPU. But both Raspberry Pi CM3 and Oculus are embedded systems with limited performance, so GPU acceleration is the only way to obtain FPS and latency needed.