Faster framerate

OpenCV, Python and other image processing questions
Post Reply
Michael93
Posts: 3
Joined: Sat Dec 14, 2019 12:12 pm

Faster framerate

Post by Michael93 »

Hello.

I am new in using stereopi and I am looking for the solution of the problem in my project in this forum :)

I have such a code written in Python that captures images from cameras :

camera = picamera.PiCamera(stereo_mode='side-by-side', sensor_mode=7, framerate=70)
camera.resolution = (320*2, 240)
time.sleep(2)
start = time.time()
stream = io.BytesIO()
for foo in camera.capture_continuous(stream, 'jpeg', burst=True):
img_counter += 1
if img_counter == 30:
FPS = (1.0 * img_counter) / (time.time() - start_time)
print("FPS: %0.2f" % FPS)
img_counter = 0
start_time = time.time()

Using this method I gather around 23 frames per second. Using Raspbian S.L.P. Image I observe the higher frequency of gathered frames? Is there any way to speed up the capturing process?

Best regards,
Michael

User avatar
Realizator
Site Admin
Posts: 900
Joined: Tue Apr 16, 2019 9:23 am
Contact:

Re: Faster framerate

Post by Realizator »

Hi Michael93,
Actually Python frames capture is a bottleneck. You can do some optimizations for this. For example, you can read this brilliant answer from the PiCamera developer here. I mean the second part of his answer concerning np.frombuffer section.

You see, we're preparing a new article now, concerning performance comparison of Python and C based code. In our C code we use piped transfer from the raspividyuv to our binary, and was able to get 88 FPS for the 1280x480 greyscale image. You can use similar approach to your Python script. Here is an an example of this pipe-based approach in this gist on a GitHub.
Eugene a.k.a. Realizator

fraserbarton
Posts: 18
Joined: Sun Oct 06, 2019 11:55 am

Re: Faster framerate

Post by fraserbarton »

Realizator wrote:
Sat Dec 28, 2019 12:34 pm
Hi Michael93,
Actually Python frames capture is a bottleneck. You can do some optimizations for this. For example, you can read this brilliant answer from the PiCamera developer here. I mean the second part of his answer concerning np.frombuffer section.

You see, we're preparing a new article now, concerning performance comparison of Python and C based code. In our C code we use piped transfer from the raspividyuv to our binary, and was able to get 88 FPS for the 1280x480 greyscale image. You can use similar approach to your Python script. Here is an an example of this pipe-based approach in this gist on a GitHub.
Hi Realizator,

My application requires that I use a capture method rather than a video method, this is because I have to toggle LEDs on and off inbetween frames so that I can get one LED on image and one LED off image per cycle, I also require a reasonably high resolution image and am currently taking all captures at 1280 x 720.

I have implemented the np.frombuffer for a camera.capture(output, 'yuv') method based on the answer from the PiCamera developer and it seems to have upped my frame rate to just over 3 FPS.

Code: Select all

import time
import picamera
import numpy as np

class MyOutput(object):
    def write(self, buf):
        # write will be called once for each frame of output. buf is a bytes
        # object containing the frame data in YUV420 format; we can construct a
        # numpy array on top of the Y plane of this data quite easily:
        y_data = np.frombuffer(
            buf, dtype=np.uint8, count=1280*720).reshape((720, 1280))
        # do whatever you want with the frame data here... I'm just going to
        # print the maximum pixel brightness:
        #print(y_data[:720, :1280].max())

    def flush(self):
        # this will be called at the end of the recording; do whatever you want
        # here
        pass

with picamera.PiCamera(
        sensor_mode=4,
        resolution='1280x720',
        framerate=40) as camera:
    time.sleep(2) # let the camera warm up and set gain/white balance
    output = MyOutput()
    while True:
        start_time = time.time()
        camera.capture(output, 'yuv')
        print("--- %s seconds ---" % (time.time() - start_time))
I am looking at the pipe-based approach and see that it based around the raspividyuv method. I am just not experienced enough at programming to turn that into something that is based around the capture method.

Is there any other ways that I can further improve my framerate without using the video port?

User avatar
Realizator
Site Admin
Posts: 900
Joined: Tue Apr 16, 2019 9:23 am
Contact:

Re: Faster framerate

Post by Realizator »

Hi Fraserbaton,
While taking a still images Pi works in a special mode, and high FPS is a problem here.
Another point is that raspividyuv actually can give you a high frame rate over pipe, but Python is unable to process this stream fast enough.

Can you please tell me the background of your task?
1. What is your target FPS and resolution?
2. If we get this FPS/resolution, are you sure your Python code will be able to process it in a real-time?
3. If real-time processing is your aim? Or you just need to record a captured images with/without flash?

JFYI: Picamera has a flash support, but just for the still port. Also here is a discussion of using it with a stereoscopic setup.
Eugene a.k.a. Realizator

fraserbarton
Posts: 18
Joined: Sun Oct 06, 2019 11:55 am

Re: Faster framerate

Post by fraserbarton »

Realizator wrote:
Fri Apr 03, 2020 1:13 pm
Hi Fraserbaton,
While taking a still images Pi works in a special mode, and high FPS is a problem here.
Another point is that raspividyuv actually can give you a high frame rate over pipe, but Python is unable to process this stream fast enough.

Can you please tell me the background of your task?
1. What is your target FPS and resolution?
2. If we get this FPS/resolution, are you sure your Python code will be able to process it in a real-time?
3. If real-time processing is your aim? Or you just need to record a captured images with/without flash?

JFYI: Picamera has a flash support, but just for the still port. Also here is a discussion of using it with a stereoscopic setup.
Hi Realizator,

I am using the StereoPi to detect bright spots where the light from a multi LED flash reflects particularly intensely from the curved shiny surfaces, at the moment I am using an array of evenly spaced sewing pins (please see the image attached). Both of the cameras on the stereoPi are spaced approximately 150 mm apart and angled inwards at some angle so that their image centres cross approximately 200 mm away.

For each capture cycle I take two exposures, the first with the LEDs on and the second with the LEDs off. The reason for this is that I subtract the off image from the on image to reduce noise and make an image in which the bright reflections of the LEDs on the sewing pins are the only prominent feature remaining in the image.

My analysis is fairly lightweight, I take a horizontal intensity strip from the centre of the image (i.e. for a 1280 x 720 capture I take a 1280 x 1 strip from the 360th row). I then run a simple scipy.find_peaks() on the intensity strip and plot the data. This process is looped and the graph is refreshed for every capture.

1. I have a preference for resolution over frame rate as this is my limiting factor for the precision of the final measurement. I would like to ideally aim for an image with a horizontal resolution of 1500 and perhaps 5-10 full capture cycles (10-20 exposures) per second if possible.
2. I have tried my python code with and without the analysis and plotting operations and have found that they are fairly negligible with respect to the time it takes to simply call the camera.capture(output, 'yuv') command. I am therefore fairly confident that my analysis can deal with a much faster capture rate.
3. Real time processing is my aim as although sensor will only take measurements of stationary objects it will need to be finely positioned by hand when it is fully housed, a reasonable capture rate makes this process considerably easier. As above 5 full cycles (10 exposures) a second would be more than enough, anything beyond this would be considered a bonus.

Please see my code below (it is a work in progress and is yet to be polished to please bear with me)

Code: Select all

#Standard python modules
import time
import picamera
import picamera.array
import numpy as np
import matplotlib.pyplot as plt
from gpiozero import LED

#Project specific modules
import scan

#Initialise the camera
camera = scan.initialise_camera()

#Begin capturing live scans and displaying them as an intensity plot
scan.plot_scan_live(camera)
This is the scan.py module:

Code: Select all

#Scan module

from gpiozero import LED
import numpy as np
import time
import picamera
import matplotlib.pyplot as plt
from scipy import signal

######################
#Variable declarations
######################

#Define LED GPIO pins
red_led_0 = LED(14)
red_led_1 = LED(15)
red_led_2 = LED(18)
red_led_3 = LED(3)
red_led_4 = LED(4)
    
#Define scan resolution
h_resolution = 1280
v_resolution = 720
h_resolution_stereo = 2 * h_resolution
#For unencoded formats:
#Horizontal resolution is rounded to nearest multiple of 32
#Vertical resolution is rounded to nearest multiple of 16

#Define number of rows to be compiled into intensity strip
strip_rows = 1

#Define minimum peak height and width for detection
minimum_peak_height = 20
minimum_peak_width = 1
minimum_peak_distance = 10

#Define matplotlib style
plt.style.use('dark_background')

#Post processing toggle
post_processing_enable = True

#Use video port toggle
video_port_enable = False

######################
#Function definitions
######################

#Enable all Red LEDs
def enable_scan_leds():
    red_led_0.on()
    red_led_1.on()
    red_led_2.on()
    red_led_3.on()
    red_led_4.on()

#Disable all Red LEDs
def disable_scan_leds():
    red_led_0.off()
    red_led_1.off()
    red_led_2.off()
    red_led_3.off()
    red_led_4.off()

#Custom output for np.frombuffer method
class MyOutput(object):
    
    def _init_(self):
        self.scan_output = np.empty(1, h_resolution_stereo, dtype=np.uint8)
    
    def write(self, buf):
        # write will be called once for each frame of output. buf is a bytes
        # object containing the frame data in YUV420 format; we can construct a
        # numpy array on top of the Y plane of this data quite easily:
        y_data = np.frombuffer(buf, dtype=np.uint8, count=h_resolution_stereo*v_resolution).reshape((v_resolution, h_resolution_stereo))
        
        self.scan_output = y_data[int(v_resolution/2):int(v_resolution/2) + strip_rows, :h_resolution_stereo]
        #print(scan_output)
        #print(scan.shape)
        

    def flush(self):
        # this will be called at the end of the recording; do whatever you want
        # here
        pass

#Initialise camera for nparray Method
def initialise_camera():
    camera = picamera.PiCamera(stereo_mode ='side-by-side', stereo_decimate=True, resolution=(h_resolution_stereo, v_resolution))
    camera.hflip = True
    camera.vflip = True
    camera.shutter_speed = 30000
    camera.awb_mode = 'off'
    camera.exposure_mode = 'off'
    #time.sleep(2)
    return camera

#Grab scan from camera
def grab_scan_buffer(camera):
    #Create y_data (np.frombuffer) to capture output
    output = MyOutput()
    
    #Enable LEDs
    enable_scan_leds()
    
    try:
        camera.capture(output, 'yuv', use_video_port=video_port_enable)
    except IOError:
        pass
    
    scan_leds_on = output.scan_output
    
    #Disable LEDs
    disable_scan_leds()
    
    try:
        camera.capture(output, 'yuv',use_video_port=video_port_enable)
    except IOError:
        pass

    scan_leds_off = output.scan_output
    
    #Subtract the LEDs off scan from the LEDs on scan to reduce noise
    scan_output = np.subtract(scan_leds_on, scan_leds_off.astype(np.int16)).clip(0, 255).astype(np.uint8)
    
    return scan_output

#A function to display the left and right scans in top-bottom format. Left is top and right is bottom
def plot_scan_live(camera):
    #Can improve efficieny of plotting function using knowledge at: https://bastibe.de/2013-05-30-speeding-up-matplotlib.html
    
    #Define pixel array for plotting purposes
    pixel_number = np.linspace(1, h_resolution, num=h_resolution)
    
    #Generate figure and axes and set size of figure
    fig, axs = plt.subplots(2,1, squeeze=True, sharex=True)
    fig.set_size_inches(12, 4)
    
    #Initialise left and right lines with random data for scan profile. Left is top and right is bottom, from perspective of sensor.
    scan_line_left, = axs[0].plot(pixel_number, np.random.rand(h_resolution), color='red')
    scan_line_right, = axs[1].plot(pixel_number, np.random.rand(h_resolution), color='red')
    
    peak_markers_left, = axs[0].plot(np.random.rand(1), np.random.rand(1), 'y|')
    peak_markers_right, = axs[1].plot(np.random.rand(1), np.random.rand(1), 'y|')
    
    #Set axis limits on the graphs
    axs[0].set_ylim(0, 265)
    axs[0].set_xlim(0, h_resolution)
    
    axs[1].set_ylim(0, 265)
    axs[1].set_xlim(0, h_resolution)
    
    
    #Show the figure (block=False disables the program break that is default for plt.show())
    plt.show(block=False)
    
    while True:
        #Grab scan data from camera
        scan_output = np.transpose(grab_scan_buffer(camera))
        scan_output_left = scan_output[:1280, 0]
        scan_output_right = scan_output[1280:, 0]
        
        #Find peaks in scan output
        scan_peaks_left, _ = signal.find_peaks(scan_output_left.flatten(), height=minimum_peak_height, width=minimum_peak_width, distance=minimum_peak_distance)
        scan_peaks_right, _ = signal.find_peaks(scan_output_right.flatten(), height=minimum_peak_height, width=minimum_peak_width, distance=minimum_peak_distance)
        
        #Refresh data for scan lines
        scan_line_left.set_ydata(scan_output_left)
        scan_line_right.set_ydata(scan_output_right)
        
        #Refresh data for left and right peaks
        peak_markers_left.set_xdata(scan_peaks_left)
        peak_markers_left.set_ydata(np.full(scan_peaks_left.shape, 260))
        
        peak_markers_right.set_xdata(scan_peaks_right)
        peak_markers_right.set_ydata(np.full(scan_peaks_right.shape, 260))
        
        #Redraw the current figure with the new data
        fig.canvas.draw()
        fig.canvas.flush_events()
I am aware that it is inefficient to turn all 5 IO on for the LEDs, I am working on a second version of the LED board in which all LEDs will be toggled on with a single GPIO.

As always I really appreciate your input.
Attachments
sewing_pins_leds_on.jpeg
sewing_pins_leds_on.jpeg (2.55 MiB) Viewed 16776 times

stereomaton
Posts: 215
Joined: Tue May 21, 2019 12:33 pm
Location: France

Re: Faster framerate

Post by stereomaton »

It's a quick answer without checking anything, but have you considered the capture_continuous function of picamera which seems to be more or less written for the kind of application you have:
https://picamera.readthedocs.io/en/rele ... continuous
Stereophotographer and hacker
Despite my quite active participation in the forum, I am not in the StereoPi team
StereoPi (v1) Standard Edition + CM3Lite module + a few cameras

fraserbarton
Posts: 18
Joined: Sun Oct 06, 2019 11:55 am

Re: Faster framerate

Post by fraserbarton »

stereomaton wrote:
Sun Apr 05, 2020 12:21 pm
It's a quick answer without checking anything, but have you considered the capture_continuous function of picamera which seems to be more or less written for the kind of application you have:
https://picamera.readthedocs.io/en/rele ... continuous
I had looked into the continuous capture function but am struggling to see how I can implement the toggling of the LEDs between shots to get my led on and led off exposures. Forgive me if this is trivial, this is the first piece of software I have written.

User avatar
Realizator
Site Admin
Posts: 900
Joined: Tue Apr 16, 2019 9:23 am
Contact:

Re: Faster framerate

Post by Realizator »

Hi Fraserbarton,
I was trying to look inside camera work logic. At this moment I found, that actually you can get information about cirrent frame capture status using GPIO, originally intended for the camera LED. Here is the discussion.
This scope requires a deep investigation, but I have a hypothesis, and would like to discuss it with you.

If we can definitely know, that camera stop receiving a frame (using mentioned approach), we can use new GPIO stage to turn on your flash. So next frame will be taken with a flash. After that, we know that "flash-on" frame is captured, and can turn off the flash for the next frame. To be clear, we need not the GPIO stage itself, but a change 1 -> 0 (i.e. change from stage "I'm getting some data" to "I finished getting data from sensor").

It means, that you can use videoport to capture a frames, and can "catch" a sensor's status change (like "frame is captured").
Of course, your "leds on" and "leds off" commands need some time to be performed (i mean transients), thus you can ignore several frames to be sure, that your leds are definitely On or Off. I think with this approach you can get definitely more FPS in comparison with the still port.

One more idea. You mention "I take a horizontal intensity strip from the centre of the image". Well, you can use camera's ROI feature to crop your region, and pass this zone only to your software path instead of your image. I used this approach in Macro 3D article. I think it will save a memory bandwidth and can help to workout this bottleneck with the transition of a big data to Python over pipe.
Eugene a.k.a. Realizator

stereomaton
Posts: 215
Joined: Tue May 21, 2019 12:33 pm
Location: France

Re: Faster framerate

Post by stereomaton »

fraserbarton wrote:
Sun Apr 05, 2020 9:32 am
I am aware that it is inefficient to turn all 5 IO on for the LEDs, I am working on a second version of the LED board in which all LEDs will be toggled on with a single GPIO.
Oops, I remembered only of the first part of this sentence and sketched a schematic. Since I drew it, I will share it nonetheless.
Here is one possible solution easily doable on breadboard:

Image

R1 and R2 were computed in E12 series to have almost symmetric on and off current/time, and still enough current to turn on the leds with a pessimistic gain of 50 of the NPN transistor assuming they consume 20mA each.
fraserbarton wrote:
Sun Apr 05, 2020 6:10 pm
I had looked into the continuous capture function but am struggling to see how I can implement the toggling of the LEDs between shots to get my led on and led off exposures. Forgive me if this is trivial, this is the first piece of software I have written.
I do not have the energy today to help you much, but I did a simple test on the StereoPi with V1 cameras with the following code:

Code: Select all

import io
import time
import picamera
import numpy as np

max_frames = 500
frames = max_frames
def process(stream):
    global frames
    frames -= 1
    stream.seek(0)
    data = np.fromstring(stream.getvalue(), dtype=np.uint8)
    l = len(data)
    if l != 1280*720*1.5: print('Error')
    return frames > 0

with picamera.PiCamera(stereo_mode ='side-by-side', stereo_decimate=True, resolution=(1280, 720)) as camera:
    camera.shutter_speed = 30000
    camera.awb_mode = 'off'
    camera.exposure_mode = 'off'
    camera.framerate=25

    stream = io.BytesIO()
    start = time.perf_counter()
    for foo in camera.capture_continuous(stream, format='yuv', use_video_port=True):
        stream.truncate()
        if process(stream):
            stream.seek(0)
        else:
            break
    elapsed = time.perf_counter() - start
    print('Elapsed:', elapsed)
    print('FPS:', max_frames/elapsed)
Notice that I had to use the video port.
With video port: ~24.8 fps
With still port: ~1.5 fps
According to the documentation, the main difference is the amount of noise reduction.

Here a crop in the image (without scaling) to appreciate the difference.
left is with video port, right is with still port ; over is direct output under low illumination, under is the same processed with equalization.
In my opinion, the difference is sufficiently tiny for your application:

Image

Notice that in YUV format (which is YUV420 here), the first W*H bytes are the Y component (i.e. "grayscale") and the two next W*H*1/4 are UV planes (i.e. "colors") which you can probably ignore.
All the processing can be done in the process() function, including changing the state of the leds.
Here I just count the frames and check the length of the image, but you can do what you need.
The numpy array is an example that might help you. If you want to get a subpart, you can probably seek to the start position and add a count= parameter in np.fromstring with the length of your cut. I tried to use the "zoom" attribute of picamera, but it did not work on the first test and I did not search more.
The function should return False to quit.

I hope those little information, while not very structured, can help you.
Stereophotographer and hacker
Despite my quite active participation in the forum, I am not in the StereoPi team
StereoPi (v1) Standard Edition + CM3Lite module + a few cameras

fraserbarton
Posts: 18
Joined: Sun Oct 06, 2019 11:55 am

Re: Faster framerate

Post by fraserbarton »

Realizator wrote:
Mon Apr 06, 2020 3:38 pm
Hi Fraserbarton,
I was trying to look inside camera work logic. At this moment I found, that actually you can get information about cirrent frame capture status using GPIO, originally intended for the camera LED. Here is the discussion.
This scope requires a deep investigation, but I have a hypothesis, and would like to discuss it with you.

If we can definitely know, that camera stop receiving a frame (using mentioned approach), we can use new GPIO stage to turn on your flash. So next frame will be taken with a flash. After that, we know that "flash-on" frame is captured, and can turn off the flash for the next frame. To be clear, we need not the GPIO stage itself, but a change 1 -> 0 (i.e. change from stage "I'm getting some data" to "I finished getting data from sensor").

It means, that you can use videoport to capture a frames, and can "catch" a sensor's status change (like "frame is captured").
Of course, your "leds on" and "leds off" commands need some time to be performed (i mean transients), thus you can ignore several frames to be sure, that your leds are definitely On or Off. I think with this approach you can get definitely more FPS in comparison with the still port.

One more idea. You mention "I take a horizontal intensity strip from the centre of the image". Well, you can use camera's ROI feature to crop your region, and pass this zone only to your software path instead of your image. I used this approach in Macro 3D article. I think it will save a memory bandwidth and can help to workout this bottleneck with the transition of a big data to Python over pipe.
Hi Realizator,

I have only just started to come back to this.

I have realised that I really need to optimize the way in which I am capturing frames before I go on to write the rest of the software.

FYI I am using the waveshare G cameras that came with the deluxe kit, which to my understanding uses the OV5647 sensor and therefore the hardware constraints of the V1 camera module apply. Correct me if I am wrong.

I am not fussy about resolution but I want to achieve a capture rate of roughly 5-10fps ideally although less is not a huge issue.

I have played around with the capture_continuous method and find that I am happy with the capture rate when using the capture_continuous method with use_video_port set to True.

Starting from basics is there any advantage in terms of frame rate and FOV between using sensor_mode 5 (https://picamera.readthedocs.io/en/rele ... .html#mmal) which I see can give me a resolution of 1296x730 using 2x2 binning compared to setting the resolution to 1280x720 when creating the picamera object.

If I use the ROI method to get the gpu to only return the horizontal strip that I am after, is it only the strip that gets written to output when calling capture_continuous? Does this have the capture resolution or the strip resolution? i.e. 1280 x 720 or 1280 x 1? Does this offer any speed advantage?

Post Reply