Raspivid 3D Top-Bottom Bug

S.L.P. image questions, stereoscopic video livestream and recording, stereoscopic photo capture etc.
Post Reply
phowe
Posts: 8
Joined: Sun Feb 28, 2021 4:40 am

Raspivid 3D Top-Bottom Bug

Post by phowe »

Hello all,

I am using the custom Raspivid file from the SLP image on my own version of Raspian Buster. I have written a python script utilizing openCV to create a blank, fullscreen window that displays sensor data from a BME280 at the top of the screen in real-time. Then, I go into terminal to startup Raspivid and the python script at the same time. When I use Raspivid with the side-by-side mode, everything runs flawlessly and as expected. However, when I run Raspivid with the top-bottom mode (which I need for the screen I am using), the video feed freezes. When just running Raspivid in top-bottom mode, or the python script by itself, there are no issues, but running them at the same time causes Raspivid to freeze.

If I enter this command with side-by-side mode, it runs great and works as expected.

Code: Select all

raspivid -3d sbs -w 1080 -h 1920 -p 68,64,1016,1792 & python ./bme_display.py
This is what the screen shows:
http://imgur.com/a/Y4rXaRE

If I enter this command for top-bottom mode, the video feed freezes.

Code: Select all

raspivid -3d tb -w 1080 -h 1920 -p 68,64,1016,1792 & python ./bme_display.py
This is what the screen shows:
http://imgur.com/ttS1vuQ

If I enter this command to just run the Raspivid in top-bottom mode, the video feed runs just fine.

Code: Select all

raspivid -3d tb -w 1080 -h 1920 -p 68,64,1016,1792
This is what the screen shows:
http://imgur.com/mQ0LsUV

I'm not sure what is causing this issue and any help would be greatly appreciated!
Last edited by phowe on Mon Apr 05, 2021 3:18 pm, edited 2 times in total.

stereomaton
Posts: 210
Joined: Tue May 21, 2019 12:33 pm
Location: France

Re: Raspivid 3D Top-Bottom Bug

Post by stereomaton »

From what I understood, the video preview is directly managed by the gpu that bypasses the display manager, displaying the video on an overlay composited by the hardware. Thus the first thing I would try in your case is to increase the video memory split size.
If it doesn't work, I would input the feed in the opencv program (realizator wrote an article about it) and place the left and right image in the top-bottom configuration by script to overcome the bug.
Stereophotographer and hacker
Despite my quite active participation in the forum, I am not in the StereoPi team
StereoPi (v1) Standard Edition + CM3Lite module + a few cameras

phowe
Posts: 8
Joined: Sun Feb 28, 2021 4:40 am

Re: Raspivid 3D Top-Bottom Bug

Post by phowe »

@stereomaton,

Thank you for the reply! I have a few questions. By increasing the video memory split size, do you mean increasing the width and height of the video feed, or the size of the preview window?

Wouldn’t passing the camera feed through opencv and converting to top-bottom view decrease performance? Every time I’ve tried to run PiCamera with openCV and python, the performance is significantly reduced when compared to native raspivid.

Any chance you have a link to the article?

stereomaton
Posts: 210
Joined: Tue May 21, 2019 12:33 pm
Location: France

Re: Raspivid 3D Top-Bottom Bug

Post by stereomaton »

I talked about gpu_mem in config.txt of the BOOT partition.

The GPU and OS share the same RAM, but cannot interact to manage the allocations, and thus we have to define where is the split, that is which amount will be let untouched by the OS so that the GPU can use it and the way around.

Since the incorrect behavior occurs only when the two programs are launched, I suppose that it might potentially be that the RAM usage of the GPU is close to the limit and a bit too much with the two programs. It is a very uncertain assumption, but easy to test (add 128Mo of video memory, reboot, and see if it solves the problem, then possibly reduce the amount if recovering some RAM for the OS is important to you)

Of course passing through a program to display the image would be slower than direct display by the GPU, but if it is an actual bug and not a memory issue, you need a workaround.
Unfortunately, I do not find his article again. In my memory, he explained that feeding raspividyuv into the opencv program via the standard input, even in python, allowed to have not-incredibly-slow framerate, but I might be wrong. Maybe could he correct me and drop the link when he will read this.
Stereophotographer and hacker
Despite my quite active participation in the forum, I am not in the StereoPi team
StereoPi (v1) Standard Edition + CM3Lite module + a few cameras

phowe
Posts: 8
Joined: Sun Feb 28, 2021 4:40 am

Re: Raspivid 3D Top-Bottom Bug

Post by phowe »

@stereomaton,

I tried changing the GPU allocation to no avail (except for finding that 128Mb is not enough to run Raspivid). However, after more testing, I have stumbled upon something interesting. I can get the video feed from the cameras to run without freezing, under certain conditions. I can only get it to work if I reboot, run Raspivid by itself first for a few seconds, and then enter the command to run both Raspivid and the python program together. If I close the program and stop Raspivid and then attempt to redo the process without rebooting, the video feed freezes once more.

However, not all is clear yet. Where there were once green bars, there are now weird artifacts that occur. Everything else runs smoothly, except in these little sections on the left side of each camera's feed. Here is a link to a video I took that showcases the weird artifacts.

https://youtu.be/2eV6lQB7_a4

I am not sure what is going on. Any thoughts?

User avatar
Realizator
Site Admin
Posts: 683
Joined: Tue Apr 16, 2019 9:23 am
Contact:

Re: Raspivid 3D Top-Bottom Bug

Post by Realizator »

Hi Phowe,
I have a few questions:
1. On your first photo I see you are using "raspivid -3d sbs -w 1080 -h 1920", but the video on your screen is top-bottom. To say briefly, with these settings you will get a stereoscopic image with 540x1920 resolution for each camera. Are you sure you want this result?

2. Am I got you right, that your HDMI screen is 1080x1920? If yes - you can look at our approach for this screen (and raspivid capture settings) in our article here.

3. As Stereomaton mentioned, raspistill is doing its screen overlay by GPU, and has a few tricks under the hood. For example, you might face the green lines if your resolution is not dividable by 32. For example, try to set 1088 instead of 1080. Usually, these issues are critical for raspividyuv (as you are getting the raw data from the GPU). Raspivid tries to fix all issues by scaling, but sometimes it fails.

4. Using preview options (-p) adding a lot of confusion to raspivid. Please note...

5. I don't know how you are adding the text to the screen. In my ancient project 3dberry.org (before the StereoPi) I used the screen overlay to put depth map data on it. I don't know if this applicable in your case, but you can take a look here. As I understand, you want to have a black background under the text. So instead of using "-p" you can try to add an overlay with the black fields.
Eugene a.k.a. Realizator

phowe
Posts: 8
Joined: Sun Feb 28, 2021 4:40 am

Re: Raspivid 3D Top-Bottom Bug

Post by phowe »

Hey Realizator,

Thanks for the reply and I have answers to your questions!

1) I used the sbs mode in the first command to showcase that the side-by-side mode has no issues with freezing. On a landscape display, I also attempted both side-by-side mode and top-bottom mode and only saw freezing when using the top-bottom mode. From these tests, I have concluded the issue is with top-bottom mode itself rather than any sort of display issue.

2) The screen is indeed 1080x1920 and is actually the same one used in the article, so I have based my approach on the article's findings and settings.

3) I attempted using 1088 instead of 1080, as well as increasing the preview window to 1024 from 1016, but neither got rid of the artifacts.

4 & 5) I am using the preview mode so that I can add text to the screen. Initially, I sought out to overlay text directly on top of the Raspivid feed, but I was entirely unsuccessful. I also tried using Picamera and openCV with their overlay features, but performance was significantly worse than running native Raspivid and would be unsuitable for my project. This led me to using preview mode as it allows me to use native Raspivid without taking up the entire screen.

In essence, I use openCV in python to create an array of zeros that takes up the entire screen and place the sensor data in the text at the top of the screen. Then I run Raspivid and the python program at the same time. Since Raspivid is displayed as the top layer, it is overlaid on top of the python program. However, since Raspivid is in preview mode and does not take up the entire screen, the sensor data displayed at the top of the screen in the python program is visible.

User avatar
Realizator
Site Admin
Posts: 683
Joined: Tue Apr 16, 2019 9:23 am
Contact:

Re: Raspivid 3D Top-Bottom Bug

Post by Realizator »

Phowe, thank you for your answers!
One more question. Are you need the synced stereoscopic image? Or you just need to use two cameras (with no exact sync)? In this case, you can use two raspivid instances, and set appropriate preview windows for them, like 6by9 recommended in this post:
https://www.raspberrypi.org/forums/view ... p?t=203418

UPD> Actually you have advanced control on the MMAL from the PiCamera module. Look at this extremely tricky and interesting approach from the PiCamera author: https://raspberrypi.stackexchange.com/q ... 0126#60126 . Please note the approach shown is used for a single camera, but inside the PiCamera you can use two cams independently, accessing them by number.
Eugene a.k.a. Realizator

phowe
Posts: 8
Joined: Sun Feb 28, 2021 4:40 am

Re: Raspivid 3D Top-Bottom Bug

Post by phowe »

Realizator,

I need the the stereoscopic image synced since I will be using this as a VR display.

User avatar
Realizator
Site Admin
Posts: 683
Joined: Tue Apr 16, 2019 9:23 am
Contact:

Re: Raspivid 3D Top-Bottom Bug

Post by Realizator »

Oh, got you. If your cameras are HQ cameras, you can sync them in a hardware manner. For other models (V1/V2/generics) this is possible too, but more tricky.
As for the TB mode - looks like its realization in raspivid is not ideal. As I understand, it works fine without your python script, but freeze if you run both raspivid and a script.
You mentioned that the camera preview window is in the top layer. But I was able to add an overlay (link to 3dberry in my previous answer) at the top of the preview. THis also mentioned in PiCamera docs, and, for example, in this question.

Some ideas:
1. Did you try to use PiCamera as an advanced tool to access MMAL and just show a preview (without image processing)? In this case preview is GPU-driven, and should not be slow.
2. Did you play with the run sequence of your code parts (raspivid first or the Python script first)?
Eugene a.k.a. Realizator

phowe
Posts: 8
Joined: Sun Feb 28, 2021 4:40 am

Re: Raspivid 3D Top-Bottom Bug

Post by phowe »

Both of my cameras are V1 and came included in the StereoPi kit. You are pretty much correct, except I did find one instance where I can get Raspivid and the script to run together (see above post in reply to stereomaton in regards to increasing the GPU memory).

In my search for overlays, I found a lot of examples for displaying an image or static text on top of the video feed or picture stills. However, I need dynamic text since I want to display the data from the BME280 sensor in real-time. I have not been able to find any concrete examples of dynamic text being overlaid, except for the built-in frame # and time features available in PiCamera. Does your overlay display dynamic text or is it static?

As for your suggestions:
1) I have not tried using PiCamera to control MMAL and create a preview, but I will attempt to do so after seeing the posts you have linked! However, I still have concerns about the level of performance and maintaining synchronization.

2) I tried reversing the order and there were no changes. I can get both to work only under the process of rebooting, running Raspivid by itself for a few seconds, closing Raspivid, and then running Raspivid and the script together. However, I still have the weird artifacts that occur where the green bars used to be.

User avatar
Realizator
Site Admin
Posts: 683
Joined: Tue Apr 16, 2019 9:23 am
Contact:

Re: Raspivid 3D Top-Bottom Bug

Post by Realizator »

In my 3dberry code I used an overlay to show the depth map on the right part of the screen. It was shown at ~20FPS in that case (as Pi2 can't do this faster at that moment) . I redraw half of the screen with the depth map image. As I understand, in your case you don't need that FPS for the sensor data. So you can convert your text to pixels and add these pixels to the overlay layer.
In our later scripts (stereopi-tutorial and stereopi-fisheye-robot) we stopped using video preview, and draw all data by Python.
Eugene a.k.a. Realizator

Post Reply