Continuous Capture in the Background

OpenCV, Python and other image processing questions
Post Reply
fraserbarton
Posts: 18
Joined: Sun Oct 06, 2019 11:55 am

Continuous Capture in the Background

Post by fraserbarton »

In my application I take a horizontal intensity profile of an array of sewing pins. My setup has a set of LEDs that are toggled on and off in between frame captures. The OFF scan is subtracted from the ON scan to reduce background noise, and therefore a full scan is only produced every two capture cycles. The resultant scan is plotted on a MatPlotLib graph.

What I would like to achieve is that the capture_continuous function runs in the background, and the graph is only updated when a full scan has been produced. I understand that a simple way to do this would be to only have the graph update when the OFF scan is captured. However, I am trying to separate the capture process from the graph plotting process.

I envision that there will eventually be a GUI where the graph is always visible, and only starts updating when you click a 'LIVE MODE' button.

My initial idea had been to have the capture_continuous process running as a multiprocessing process and set a boolean to True every time it had produced a full scan. On top of this the graph would be continuously checking the state of the boolean so that it would update every time the boolean was set to True, set it back to False and update the graph. I am not having much luck with this as the capture_continuous process doesn't seem to run when it is run as a multiprocessing process, meaning that the boolean always stays as False and the graph never updates. I have confirmed this using a remote_pdb breakpoint. The capture_continuous process seems to run fine when it is just called as a function as I can see that the LEDs rapidly toggle state.

My questions are:

Am I doing something wrong?
Is there a better way to achieve this? I have seen something called the observer pattern that seems like it might be appropriate?

Code: Select all

from gpiozero import LED
import numpy as np
from time import sleep
import picamera
import matplotlib.pyplot as plt
from scipy import signal
import csv
import os
from multiprocessing import Process, current_process
from remote_pdb import RemotePdb

#LED Pins
LED_STATUS_RED = 18
LED_STATUS_GREEN = 15
LED_STATUS_BLUE = 14
LED_SCAN_0 = 21
LED_SCAN_1 = 20
LED_SCAN_2 = 16
LED_SCAN_TOGGLE = 19

# Scan resolution
H_RESOLUTION = 1280
V_RESOLUTION = 32
H_RESOLUTION_STEREO = 2 * H_RESOLUTION

# Set plot style
plt.style.use('dark_background')

sensor = Sensor()

live_mode(sensor)

def live_mode(sensor):
        process = Process(target=sensor.grab_scan_continuous, args=(), daemon=True)
        process.start()
        while True:
            print(sensor.scan_available)
            if sensor.scan_available:
                self.plot.update_scan_only(self.sensor.current_scan_left, self.sensor.current_scan_right)

class Sensor():
    def __init__(self):
        # SHS configuration
        self.camera = picamera.PiCamera(stereo_mode ='side-by-side', stereo_decimate=False, resolution=(H_RESOLUTION_STEREO, V_RESOLUTION))
        self.camera.awb_mode = 'off'
        self.camera.exposure_mode = 'off'
        self.camera.shutter_speed = 10000
        self.video_port_enable = True
        # Scan LEDs
        self.scan_leds = ScanLED()
        # SensorOutput object
        self.output = SensorOutput()
        #Current scan
        self.current_scan_left = np.zeros(H_RESOLUTION)
        self.current_scan_right = np.zeros(H_RESOLUTION)
        #Scan available flag
        self.scan_available = False

    def grab_scan_continuous(self):
        #RemotePdb('127.0.0.1', 4444).set_trace()
        self.scan_leds.toggle.on()
        for frame in self.camera.capture_continuous(self.output, format='yuv', use_video_port=self.video_port_enable):
            RemotePdb('127.0.0.1', 4444).set_trace()
            if self.scan_leds.toggle.is_lit:
                scan_leds_on = self.output.scan
                self.scan_leds.toggle.off()
            else:
                scan_leds_off = self.output.scan
                scan_data = np.transpose(np.subtract(scan_leds_on, scan_leds_off.astype(np.int16)).clip(0, 255).astype(np.uint8))
                self.current_scan_left = scan_data[:H_RESOLUTION]
                self.current_scan_right = scan_data[H_RESOLUTION:]
                #print(signal.find_peaks(scan_data.flatten(), minimum_peak_height, minimum_peak_width, minimum_peak_distance))
                self.scan_available = True
                self.scan_leds.toggle.on()
                
 class Plot:
    def __init__(self):
        #Can improve efficieny of plotting function using knowledge at: https://bastibe.de/2013-05-30-speeding-up-matplotlib.html
        #Define pixel array for plotting purposes
        self.pixel_number = np.linspace(1, H_RESOLUTION, num=H_RESOLUTION)
        #Generate figure and axes and set size of figure
        self.fig, self.axs = plt.subplots(2,1, squeeze=True, sharex=True)
        self.fig.set_size_inches(12, 4)
        #Initialise left and right scan lines with random data for scan profile. Left is top and right is bottom, from perspective of sensor.
        self.scan_data_left, = self.axs[0].plot(self.pixel_number, np.random.rand(H_RESOLUTION), color='red')
        self.scan_data_right, = self.axs[1].plot(self.pixel_number, np.random.rand(H_RESOLUTION), color='red')
        #Initialise left and right master profile line with random data. Left is top and right is bottom, from perspective of sensor.
        #self.master_profile_left, = self.axs[0].plot(self.pixel_number, np.random.rand(H_RESOLUTION), color='silver')
        #self.master_profile_right, = self.axs[1].plot(self.pixel_number, np.random.rand(H_RESOLUTION), color='silver')
        #Initialise left and right peak markers with random data. Left is top and right is bottom, from perspective of sensor.
        self.peak_markers_left, = self.axs[0].plot(np.random.rand(1), np.random.rand(1), 'y|')
        self.peak_markers_right, = self.axs[1].plot(np.random.rand(1), np.random.rand(1), 'y|')
        #Set axis limits on the graphs
        self.axs[0].set_ylim(0, 265)
        self.axs[0].set_xlim(0, H_RESOLUTION)
        self.axs[1].set_ylim(0, 265)
        self.axs[1].set_xlim(0, H_RESOLUTION)
        #Show the figure (block=False disables the program break that is default for plt.show())
        plt.show(block=False)

    def update_scan_only(self, scan_data_left, scan_data_right):
        #Refresh data for scan lines
        self.scan_data_left.set_ydata(scan_data_left)
        self.scan_data_right.set_ydata(scan_data_right)
        #Redraw the current figure with the new data
        self.fig.canvas.draw()
        self.fig.canvas.flush_events()       

class SensorOutput(object):
    def _init_(self):
        self.scan = np.empty(1, H_RESOLUTION_STEREO, dtype=np.uint8)
    
    def write(self, buf):
        # write will be called once for each frame of output. buf is a bytes
        # object containing the frame data in YUV420 format; we can construct a
        # numpy array on top of the Y plane of this data quite easily:
        y_data = np.frombuffer(buf, dtype=np.uint8, count=H_RESOLUTION_STEREO*V_RESOLUTION).reshape((V_RESOLUTION, H_RESOLUTION_STEREO))
        self.scan = y_data[0, :H_RESOLUTION_STEREO]
            
    def flush(self):
        # this will be called at the end of the recording; do whatever you want
        # here
        pass
        
 class ScanLED:
    def __init__(self):
        # Define scan LED pins
        self.top = LED(LED_SCAN_0)
        self.middle = LED(LED_SCAN_1)
        self.bottom = LED(LED_SCAN_2)
        self.toggle = LED(LED_SCAN_TOGGLE)
        self.top.on()
        self.middle.on()
        self.bottom.on()

stereomaton
Posts: 215
Joined: Tue May 21, 2019 12:33 pm
Location: France

Re: Continuous Capture in the Background

Post by stereomaton »

Is this the actual code ?
By reading it, it looks like there are syntax errors, problem with the place where sensor variable is created (before declaration of its class) and that even after this is corrected, the code will do nothing interesting as is (except probably turn the LEDs on) because only the init code is called.

Multitasking is not a trivial way to program, and this usually requires some forms of synchronizations more robust than checking a common unprotected global variable. The Observer pattern works well in GUI but will probably not be helpful in this context where you have a blocking call.
I am not an expert in python, but it seems that the queue (https://docs.python.org/3/library/queue.html) object allows to exchange data in a producer/consumer way in multithreaded program with appropriate locks to protect against race conditions. Probably something that you might find useful in your case.

Toy example:

Code: Select all

import threading, queue, time

q_cmd = queue.Queue()
q_data = queue.Queue()

def fake_frame_grabber():
    i = 0
    run = True
    play = False
    while run:
        try:
            cmd = q_cmd.get(block=(not play))
            if cmd == 'start':
                play = True
            elif cmd == 'stop':
                play = False
            elif cmd == 'quit':
                play = False
                run = False
            q_cmd.task_done()
        except queue.Empty:
            pass

        if play:
            time.sleep(2)
            q_data.put(i)
            i += 1

def simulate_user_input():
    time.sleep(3)
    q_cmd.put('start')
    print('start')
    time.sleep(5)
    q_cmd.put('stop')
    print('stop')
    time.sleep(8)
    q_cmd.put('start')
    print('start')

ffg = threading.Thread(target=fake_frame_grabber)
ffg.start()

starter = threading.Thread(target=simulate_user_input)
starter.start()

while True:
    try:
        item = q_data.get(timeout=0.5)
        print('Got', item)
        q_data.task_done()
        if item == 10:
            q_cmd.put('quit')
            break
    except queue.Empty:
        print('Timeout work') # If you want to do nothing, use pass and remove the timeout on q_data.get

q_cmd.join()
starter.join()
ffg.join()
Stereophotographer and hacker
Despite my quite active participation in the forum, I am not in the StereoPi team
StereoPi (v1) Standard Edition + CM3Lite module + a few cameras

fraserbarton
Posts: 18
Joined: Sun Oct 06, 2019 11:55 am

Re: Continuous Capture in the Background

Post by fraserbarton »

Hi steromaton,

Apologies, this is not the actual code. I tried to break it down into a smaller example of the core processes so that it wouldn't be such a pain for someone else to have a look over.

I understand that multiprocessing is not a trivial thing to achieve and for this reason I would probably be best to avoid it. Looking online however I cannot see an obvious way as to how I could achieve this other than through multiprocessing.

In terms of the problem itself, do you feel that there are simpler ways to achieve this other than using multiprocessing?
stereomaton wrote:
Sat Oct 24, 2020 9:20 pm
Is this the actual code ?
By reading it, it looks like there are syntax errors, problem with the place where sensor variable is created (before declaration of its class) and that even after this is corrected, the code will do nothing interesting as is (except probably turn the LEDs on) because only the init code is called.

Multitasking is not a trivial way to program, and this usually requires some forms of synchronizations more robust than checking a common unprotected global variable. The Observer pattern works well in GUI but will probably not be helpful in this context where you have a blocking call.
I am not an expert in python, but it seems that the queue (https://docs.python.org/3/library/queue.html) object allows to exchange data in a producer/consumer way in multithreaded program with appropriate locks to protect against race conditions. Probably something that you might find useful in your case.

Toy example:

Code: Select all

import threading, queue, time

q_cmd = queue.Queue()
q_data = queue.Queue()

def fake_frame_grabber():
    i = 0
    run = True
    play = False
    while run:
        try:
            cmd = q_cmd.get(block=(not play))
            if cmd == 'start':
                play = True
            elif cmd == 'stop':
                play = False
            elif cmd == 'quit':
                play = False
                run = False
            q_cmd.task_done()
        except queue.Empty:
            pass

        if play:
            time.sleep(2)
            q_data.put(i)
            i += 1

def simulate_user_input():
    time.sleep(3)
    q_cmd.put('start')
    print('start')
    time.sleep(5)
    q_cmd.put('stop')
    print('stop')
    time.sleep(8)
    q_cmd.put('start')
    print('start')

ffg = threading.Thread(target=fake_frame_grabber)
ffg.start()

starter = threading.Thread(target=simulate_user_input)
starter.start()

while True:
    try:
        item = q_data.get(timeout=0.5)
        print('Got', item)
        q_data.task_done()
        if item == 10:
            q_cmd.put('quit')
            break
    except queue.Empty:
        print('Timeout work') # If you want to do nothing, use pass and remove the timeout on q_data.get

q_cmd.join()
starter.join()
ffg.join()

stereomaton
Posts: 215
Joined: Tue May 21, 2019 12:33 pm
Location: France

Re: Continuous Capture in the Background

Post by stereomaton »

If I understand well, you want to have a part with blocking calls, and another independent that can respond during this, through probably another blocking call for the interface.

There are many ways to deal with this. One classical way is to use an event loop (but your blocking calls have to be compatible with the event loop mechanism, typically expose their file descriptor and have a way to react to the unblocked event), be sure that you do not have other blocking or long processing calls and think in terms of events which is not always easy depending on the application; this is close to the observer pattern but with more engine logic, and is very good for gui logic. Another way is multitasking (with lots of variants), but you have to take extra care in the interactions between the tasks, which can be threads (same program) or processes (separate programs).

Being in the same program offers more flexibility and more ways to do it wrong. In your case, using queues is probably the way to go in this situation.
If the communication is unidirectional, it might be easier, or at least less error-prone, to use two different processes linked with a pipe. This pipe can be created in the shell (classical | character) or via Popen in python. The shell variant is the easiest since you just write to stdout in the producer (print) and read input in the consumer, and you can easily test independently. Pipes are really powerful.
If you have a bidirectionnal communication, you can have similar principle with Popen used in bidirectional way, but I would suggest to use sockets instead.There are probably libraries made to transport high level messages over such channel, but I do not know them.
There are probably other ways, but you already have some choices.

In your case, I would probably go for piping two programs because it is super easy and powerful for simple cases; but for a more interactive user interface, I would probably go for sockets unless there were tons of data (e.g. a stream of images/results) to transfer in which case I would opt for multithreading. But other architectures and choices might be valid as well.
Stereophotographer and hacker
Despite my quite active participation in the forum, I am not in the StereoPi team
StereoPi (v1) Standard Edition + CM3Lite module + a few cameras

fraserbarton
Posts: 18
Joined: Sun Oct 06, 2019 11:55 am

Re: Continuous Capture in the Background

Post by fraserbarton »

If I understand well, you want to have a part with blocking calls, and another independent that can respond during this, through probably another blocking call for the interface
Could you elaborate what you mean by blocking call in this context? To my understanding they are to do with pausing execution until an input is received?

I find your idea to use sockets interesting, this way I could separate the operation of the sensor from the software that works on the scans it produces.

The communication would need to be bidirectional I believe, as I wish to be able to control the operation of the sensor from the other piece of software, e.g. send a single scan back, start sending scans continuously, stop sending scans continuously.

stereomaton
Posts: 215
Joined: Tue May 21, 2019 12:33 pm
Location: France

Re: Continuous Capture in the Background

Post by stereomaton »

What I mean by a blocking call is a call to a method that place the task in a blocking state in the operating system, which means that the task will not consume CPU time until the appropriate event occurres (may be a timeout, a user input, the answer of a peripheral, the release of a mutex by another task, the reception of a network paquet, etc. depending on what the method waits for). In the case of a graphical user interface, there is often a call to a method that keep executing using blocking calls internally to manage its event loop.

Sending commands and receiving data is indeed bidirectional. Using sockets in your case is probably a good idea. As a side effect, you could have the interface on a computer and the measuring system on the stereopi headless if you wish.
Stereophotographer and hacker
Despite my quite active participation in the forum, I am not in the StereoPi team
StereoPi (v1) Standard Edition + CM3Lite module + a few cameras

fraserbarton
Posts: 18
Joined: Sun Oct 06, 2019 11:55 am

Re: Continuous Capture in the Background

Post by fraserbarton »

On a side note do you have any idea what would be a good starting point to work from in terms of designing an interface between the two pieces of software, one being the slave software that listens for requests for images and sends them back, the other being the master software that requests the images from the sensor software.

As the functionality required is likely to be limited, I was thinking of something simple such as numbers?

E.G.

Number Sent Action taken by Slave
1 Send single image
2 Send images continuously
3 Stop sending images continuously
4 Manually adjust exposure value of cameras

etc.

Is there a better more standardised technique to designing a simple but functional interface between two devices?

User avatar
Realizator
Site Admin
Posts: 900
Joined: Tue Apr 16, 2019 9:23 am
Contact:

Re: Continuous Capture in the Background

Post by Realizator »

Hello Fraserbaton and Stereomaton,
Please let me jump into your discussion :-)
In my point 1 of this answer, I attached a multithreaded Python code for the capture_continuous, and also added a link to the original related work at Pyimagesearch. Can this help you?..
Eugene a.k.a. Realizator

fraserbarton
Posts: 18
Joined: Sun Oct 06, 2019 11:55 am

Re: Continuous Capture in the Background

Post by fraserbarton »

Realizator wrote:
Mon Oct 26, 2020 1:06 pm
Hello Fraserbaton and Stereomaton,
Please let me jump into your discussion :-)
In my point 1 of this answer, I attached a multithreaded Python code for the capture_continuous, and also added a link to the original related work at Pyimagesearch. Can this help you?..
Hi Realizator,

In the context of continuous_capture, is threading more appropriate than multiprocessing?

Would threading/multiprocessing give me an advantage as opposed to using sockets?

User avatar
Realizator
Site Admin
Posts: 900
Joined: Tue Apr 16, 2019 9:23 am
Contact:

Re: Continuous Capture in the Background

Post by Realizator »

Fraserbarton, I can't say what is more appropriate in your case. Unfortunately, I'm not an expert in multiprocessing...
Ideally, you should find a bidirectional way to exchange data between two processes. If you don't need a low-latency, real-time data exchange, you can try a "noob" idea and use a file for the exchange. Both master and slave will periodically read it, and try to record new data (i.e. Master write commands, Slave reads and put a result). Ofcourse, they should check, either the file is available for recording, or another process did not release it yet.
For the real-time data exchange between our software parts (for example, in the SLP image), we are using network exchange. Each piece of software sits on its own socket (at IP 127.0.0.1), and sends data to other modules. But this network part might do your Python code heavyweight. We are doing this for C++ applications only.
Eugene a.k.a. Realizator

stereomaton
Posts: 215
Joined: Tue May 21, 2019 12:33 pm
Location: France

Re: Continuous Capture in the Background

Post by stereomaton »

Using multithreading (instead of sockets in this case) allows direct access to the memory of the other task. This might be useful if you want to access large memory or complex structures, but it comes with the need to protect correctly the access to any (modifiable) shared resources. Otherwise there are often strange bugs that are extremely hard to identify and debug.
Using sockets (instead of multithreading) allows to isolate the two programs by reducing the interaction to the communication channel, allows to optionally run the two programs in separate machines, allows to optionally have different interfaces with the same data producer.
Using a file to exchange data interactively is a bad idea, especially because there are more efficient ways to do the same thing better (socket, fifo, ...) without the disadvantages (such as disk usage, need to manage writing buffers, complex synchronization, ...).

I would not agree that using sockets make python code heavyweight, especially with only one client. It would be quite similar to the code needed to manage queue communication in multithreading, except that Queue is a high level interface that allow to send/copy full objects while sockets only know byte streams.

For the protocol, the easiest way if you do not use a high level library (as I said, I do not know them) is to use fixed size messages. When you read your socket, you cut the received bytes into chunks of this size and process them one after the other. If you start to need large chucks, it is better to search for a library which will probably handle variable size messages internally. Having one byte chunk for commands is a good idea if you have only less than 256 commands without arguments.
However, you said 'requests the image'. If you want to exchange image stream, the communication starts to be heavy and it is probably not a good way to go. I understood that you wanted to transfer the result of a processing, being a few numbers used to draw a graph. In this case, a fixed size message containing all the numbers computed at a given step would work, and the struct module of python could help you to construct and decode the chunk. If you really want to transfer images, the multithread way will be more appropriate.
Stereophotographer and hacker
Despite my quite active participation in the forum, I am not in the StereoPi team
StereoPi (v1) Standard Edition + CM3Lite module + a few cameras

fraserbarton
Posts: 18
Joined: Sun Oct 06, 2019 11:55 am

Re: Continuous Capture in the Background

Post by fraserbarton »

Do you think that it is multithreading that would be appropriate in this case rather than multiprocessing?

I think I favour either of those as opposed to sockets and having to operate through an interface. Much easier to interact with objects directly.

stereomaton
Posts: 215
Joined: Tue May 21, 2019 12:33 pm
Location: France

Re: Continuous Capture in the Background

Post by stereomaton »

It is up to you, I exposed some pros and cons of the different architectures. Notice that it is not because I personally do not know (and did not search for) messaging libraries that your are stuck to not use more elaborated objects in communications via sockets.

That said, I do not see what you place behind the terms multithreading and multiprocessing. With my terminology, the architecture using sockets is an example of multiprocessing (different programs collaborating in a job), while multithreading refers more specifically to a single program executing different tasks in parallel. But there might be other valid meanings of these terms that I do not think of now (thus I cannot answer)
Stereophotographer and hacker
Despite my quite active participation in the forum, I am not in the StereoPi team
StereoPi (v1) Standard Edition + CM3Lite module + a few cameras

fraserbarton
Posts: 18
Joined: Sun Oct 06, 2019 11:55 am

Re: Continuous Capture in the Background

Post by fraserbarton »

As I understand it there are two modules in python, multiprocessing and multithreading, the first allow parallel execution and the second allows concurrent.

When I originally asked the question I had looked at the two and thought that multiprocessing was more appropriate in this circumstance, although now I am not so sure.

I was really after your opinion as to which is more appropriate given the nature of the capture_continuous function?

stereomaton
Posts: 215
Joined: Tue May 21, 2019 12:33 pm
Location: France

Re: Continuous Capture in the Background

Post by stereomaton »

Okay. I just had a quick overview of these two modules and the distinction is essentially the same as I said.
One (threading) provides an interface to execute the tasks in the same process/program with shared memory, and thus you need to protect access on modifiable shared memory with appropriate mechanisms (mutexes, queues, ...)
The other (multiprocessing) provides an interface to execute the tasks in separate processes/programs, essentially through a fork system call internally which "clones" the current program and then execute a specific function in the copy. As a result you have two distinct processes with distinct memory, as if you launched two programs (but in some circumstances, having the two in the same executable is pertinent/useful/convenient), for which you need to add specific inter-process communication (such as sockets, fifo, ...) to exchange data between them.
With this precision, you can refer to the previous discussion for your choice.

If you choose threading, keep in mind what data are used by which thread [thus in which thread are executed all functions] and protect the common accesses appropriately.
If you choose to separate in different processes, you need an interface between the two parts and I would encourage to write two separate programs in your case, because it will probably be more convenient.
Stereophotographer and hacker
Despite my quite active participation in the forum, I am not in the StereoPi team
StereoPi (v1) Standard Edition + CM3Lite module + a few cameras

Post Reply