Size and framerate

OpenCV, Python and other image processing questions
Post Reply
asdf
Posts: 4
Joined: Mon May 30, 2022 3:51 pm

Size and framerate

Post by asdf »

I'm trying to capture high resolution frames from the Stereopi, reading the output of raspividyuv as suggested in this medium post. However, the framerate is terribly slow at 3264 x 1024 (1632 x 1024 per side). This is the code I'm running:

Code: Select all

import sys
import numpy as np
import cv2
import time

height = 1024
width = 1632 * 2
n_channels = 3

n_frames = 10
t0 = time.time()
for _ in range(n_frames):
    start_time = time.time()
    bytes = sys.stdin.buffer.read(width * height * n_channels)
    frame = np.frombuffer(bytes, dtype=np.uint8, count=width * height * n_channels).reshape(height, width, n_channels)
    left_frame = frame[:, :width // 2]
    right_frame = frame[:, width // 2:]
    print("--- %s seconds ---" % (time.time() - start_time))

dt = time.time() - t0
print("Framerate is %f" % (n_frames / dt))

The execution line is

Code: Select all

raspividyuv -3d sbs --rgb -w 3264 -h 1024 -fps 20 -t 0 -n -o - | python main.py

And the output is

Code: Select all

--- 0.08326101303100586 seconds ---
--- 0.09539151191711426 seconds ---
--- 0.09400773048400879 seconds ---
--- 0.09419560432434082 seconds ---
--- 0.09409403800964355 seconds ---
--- 0.09410309791564941 seconds ---
--- 0.09404730796813965 seconds ---
--- 0.09309720993041992 seconds ---
--- 0.09331631660461426 seconds ---
--- 0.0949711799621582 seconds ---
Framerate is 10.729000

As you can see, the code doesn't do anything with the images, not even using cv2.imshow()

Is this the correct approach to getting such big images? I wasn't even able to use Picamera with such resolution. I know you also can use the mmal api directly, but I wasn't able to find any examples of that.

User avatar
Realizator
Site Admin
Posts: 900
Joined: Tue Apr 16, 2019 9:23 am
Contact:

Re: Size and framerate

Post by Realizator »

Hi asdf,

You can face a few botlenecks with high resolutions, that's truth. To localize your bottleneck I'd start with checking of the real raspividyuv output FPS first before piping to Python. You can use display output to check, if you are getting 20 FPS requested or lower.
The next step is checking Pythin part. Are you going to use the whole YUV image or just the greyscale part (using 'luma" option) to save memory bandwidth?

What's your final aim? Is this just a processing, or saving/livestreaming the result? And what's the target resolution/FPS?
Eugene a.k.a. Realizator

asdf
Posts: 4
Joined: Mon May 30, 2022 3:51 pm

Re: Size and framerate

Post by asdf »

Thank you for your quick response. I did what you recommended and, indeed, it looks like Python is the bottleneck here, as a program written in C is way quicker than my Python script.

Code: Select all

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <time.h>


int main() {

    char *buffer = malloc(1024 * 1632 * 2 * 3);


    clock_t start_time = clock();
    int i;
    int n_frames = 10;
    int n_bytes;
    for (i = 0; i < n_frames; i = i + 1) {
        clock_t begin = clock();
        n_bytes = 0;
        while (1) {
            n_bytes = n_bytes + read(0, &buffer[n_bytes], 1024 * 1632 * 2 * 3 - n_bytes);
            if (n_bytes >= 1024 * 1632 * 2 * 3)
                break;
        }
        clock_t end = clock();
        double time_spent = (double)(end - begin) / CLOCKS_PER_SEC;
        printf("--- %f seconds ---\n", time_spent);
    }

    clock_t end_time = clock();

    double framerate = n_frames / ((double)(end_time - start_time) / CLOCKS_PER_SEC);

    printf("Framerate is %f\n", framerate);
    return 0;
}
I compile and run it:

Code: Select all

gcc main.c -o main
raspividyuv -3d sbs  --rgb -w 3264 -h 1024 -fps 20 -t 0 -n -o - | ./main
And the output is

Code: Select all

--- 0.097859 seconds ---
--- 0.034125 seconds ---
--- 0.033267 seconds ---
--- 0.034116 seconds ---
--- 0.033378 seconds ---
--- 0.032724 seconds ---
--- 0.033397 seconds ---
--- 0.033339 seconds ---
--- 0.032983 seconds ---
--- 0.034238 seconds ---
Framerate is 24.986133
As for your questions, I want to capture and stream images to a beefier machine which would do some processing, and I need them to be color, as the algorithms I'm using do not work with black and white images.

User avatar
Realizator
Site Admin
Posts: 900
Joined: Tue Apr 16, 2019 9:23 am
Contact:

Re: Size and framerate

Post by Realizator »

asdf, I got your aim.
If you'll forward all "raw" images to your beefier PC over network, this will be the next bottleneck. The network bandwidth to be taken into account for the RAW video stream.

You can't use H264 encoding to do this (resolution exceeds RPi GPU encoder limit), but can try MJPEG. With both encodings you'll have a data loss (tiny details). I don't know your image perfection requirements, but maybe the highest MJPEG quality and maximum bitrate (like 50 Mbit) can help you (and allow using raspivid instead of raspiyuv).

Another "tricky" way is to use... HDMI output. You can use HDMI capture device to get HDMI output to your computer. But this should be 4K-enabled HDMI capture, which is not cheap.

I like this kind of tasks - when you are working at the maximum of hardware abilities. :-) Remember you can overclock GPU encoder part if you face it's limit (heatsink is a must in this case).
Eugene a.k.a. Realizator

Post Reply