Tutorial – Python MJPEG Video Streaming on Raspberry Pi with Overlays

This article talks about how to stream video in MJPEG/MJPG format from your Raspberry Pi using Pure Python and openCV.   It supports a frame rate > 24FPS and allows overlays and information to be added on a frame by frame basis.

This software is used the the new SkyWeather system to stream live video to the WeatherSTEM cloud for use by the public.

SkyWeather  allows you to build your own weather station with a Sky Cam to take pictures of your Sky and display them the cloud on WeatherSTEM.  You might even make it to the WeatherChannel!  

• Partnership Announcement
• SkyWeather Product Announcement

All up-to-date information for SkyWeather is here.

What is MJPEG

Motion JPEG (M-JPEG or MJPEG) is a video compression format in which each video frame or interlaced field of a digital video sequence is compressed separately as an individual JPEG image. Originally developed for multimedia PC applications, M-JPEG is now used by video-capture devices such as digital cameras, IP cameras, and webcams, as well as by non-linear video editing systems. It is natively supported by the QuickTime Player, the PlayStation console, and web browsers such as Safari, Google Chrome, Mozilla Firefox and Microsoft Edge.

This format works really well for applications that want to be able to easily process each frame on the fly, versus a compressed format like mp4 or mpeg which needs to be decompressed on the fly to be manipulated by the Raspberry Pi.

MJPEG Video Streaming in Pure Python

This software gives an example of how to stream MJPEG/MJPG video on the Raspberry Pi using the PiCamera library using overlays. This is the same software that is being used in the SwitchDoc Labs SkyWeather product in conjunction with WeatherSTEM.

It  is designed to do Python video streaming in thread on SkyWeather, with single picture capability and the ability to add overlays / information on the fly.

This software requires a Rapsberry Pi 3B+ or greater.

Software Contents

streamtest.py

This Python program streams video using the openCV and the PiCamera library. Each frame is intercepted and various overlays and information is placed on the frame then sent to the video stream.

Defaults to streaming on:

http://localhost:443/

cvgrab.py

Grabs a single frame from the MJPEG video stream using openCV. Writes to a file “test.jpg”

GitHub Repository

You can find this software here:

https://github.com/switchdoclabs/SDL_Pi_MJPEGStream

Software Description

The streamtest.py software is pretty straight forward.  There are three main sections to the streamtest.py software.

Streaming Handler

class StreamingHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == '/':
            self.send_response(301)
            self.send_header('Location', '/index.html')
            self.end_headers()
        elif self.path == '/index.html':
            content = PAGE.encode('utf-8')
            self.send_response(200)
            self.send_header('Content-Type', 'text/html')
            self.send_header('Content-Length', len(content))
            self.end_headers()
            self.wfile.write(content)
        elif self.path == '/stream.mjpg':
            self.send_response(200)
            self.send_header('Age', 0)
            self.send_header('Cache-Control', 'no-cache, private')
            self.send_header('Pragma', 'no-cache')
            self.send_header('Content-Type', 'multipart/x-mixed-replace; boundary=FRAME')
            self.end_headers()
            try:
                while True:
                    with output.condition:
                        output.condition.wait()
                        frame = output.frame
                        # now add timestamp to jpeg
                        # Convert to PIL Image
                        cv2.CV_LOAD_IMAGE_COLOR = 1 # set flag to 1 to give colour image
                        npframe = np.fromstring(frame, dtype=np.uint8)
                        pil_frame = cv2.imdecode(npframe,cv2.CV_LOAD_IMAGE_COLOR)
                        #pil_frame = cv2.imdecode(frame,-1)
                        cv2_im_rgb = cv2.cvtColor(pil_frame, cv2.COLOR_BGR2RGB)
                        pil_im = Image.fromarray(cv2_im_rgb)

                        draw = ImageDraw.Draw(pil_im)

                        # Choose a font
                        font = ImageFont.truetype("/usr/share/fonts/truetype/freefont/FreeSans.ttf", 25)
                        myText = "SkyWeather "+dt.datetime.now().strftime('%Y-%m-%d %H:%M:%S')

                        # Draw the text
                        color = 'rgb(255,255,255)'
                        #draw.text((0, 0), myText,fill = color, font=font)

                        # get text size
                        text_size = font.getsize(myText)

                        # set button size + 10px margins
                        button_size = (text_size[0]+20, text_size[1]+10)

                        # create image with correct size and black background
                        button_img = Image.new('RGBA', button_size, "black")
                        
                        #button_img.putalpha(128)
                        # put text on button with 10px margins
                        button_draw = ImageDraw.Draw(button_img)
                        button_draw.text((10, 5), myText, fill = color, font=font)

                        # put button on source image in position (0, 0)

                        pil_im.paste(button_img, (0, 0))
                        bg_w, bg_h = pil_im.size 
                        # WeatherSTEM logo in lower left
                        size = 64
                        WSLimg = Image.open("WeatherSTEMLogoSkyBackground.png")
                        WSLimg.thumbnail((size,size),Image.ANTIALIAS)
                        pil_im.paste(WSLimg, (0, bg_h-size))

                        # SkyWeather log in lower right
                        SWLimg = Image.open("SkyWeatherLogoSymbol.png")
                        SWLimg.thumbnail((size,size),Image.ANTIALIAS)
                        pil_im.paste(SWLimg, (bg_w-size, bg_h-size))

                        # Save the image
                        buf= StringIO.StringIO()
                        pil_im.save(buf, format= 'JPEG')
                        frame = buf.getvalue()
                    self.wfile.write(b'--FRAME\r\n')
                    self.send_header('Content-Type', 'image/jpeg')
                    self.send_header('Content-Length', len(frame))
                    self.end_headers()
                    self.wfile.write(frame)
                    self.wfile.write(b'\r\n')
            except Exception as e:
                traceback.print_exc()
                logging.warning(
                    'Removed streaming client %s: %s',
                    self.client_address, str(e))
        else:
            self.send_error(404)
            self.end_headers()

The function has three sections in the mpeg stream. First you manipulate the jpeg file from the mjpeg stream to a numpy image and then to a PIL image to work on.

                        frame = output.frame
                        # now add timestamp to jpeg
                        # Convert to PIL Image
                        cv2.CV_LOAD_IMAGE_COLOR = 1 # set flag to 1 to give colour image
                        npframe = np.fromstring(frame, dtype=np.uint8)
                        pil_frame = cv2.imdecode(npframe,cv2.CV_LOAD_IMAGE_COLOR)
                        #pil_frame = cv2.imdecode(frame,-1)
                        cv2_im_rgb = cv2.cvtColor(pil_frame, cv2.COLOR_BGR2RGB)
                        pil_im = Image.fromarray(cv2_im_rgb)

Next you manipulate the PIL image by adding information, overlays, images, etc.

Then finally, you convert it back to a jpeg file and send it off to the video stream. The resulting image looks like this in SkyWeather:

This gives us great flexibility to add overlays and more data as the situation requires.

Streaming Service

 

class StreamingServer(SocketServer.ThreadingMixIn, HTTPServer):
    allow_reuse_address = True
    daemon_threads = True

#with picamera.PiCamera(resolution='640x480', framerate=24) as camera:
#with picamera.PiCamera(resolution='1920x1080', framerate=24) as camera:
with picamera.PiCamera(resolution='1296x730', framerate=24) as camera:
    output = StreamingOutput()
    camera.start_recording(output, format='mjpeg')
    camera.annotate_foreground = picamera.Color(y=0.2,u=0, v=0)
    camera.annotate_background = picamera.Color(y=0.8, u=0, v=0)
    try:
        address = ('', 443)
        server = StreamingServer(address, StreamingHandler)
        server.serve_forever()
    finally:
        camera.stop_recording()

 

Frame Grabbing Software

The cvgrab.py software used to grab individual frames is very short:

import cv2
print "starting grab"
cap = cv2.VideoCapture('http://localhost:443/stream.mjpg')

ret, frame = cap.read()
print "found frame"
#cv2.imshow('Video', frame)
cv2.imwrite("test.jpg",frame)
print "done"
cap.release()
print "after release"

 

Known Issues

We have been running this software for weeks with no significant issues.   One issue that we have run into with the cvgrab software is that the streaming software throws a handled exception each time a frame is grabbed, which while it doesn’t kill the streamer, it prints unwanted data printed to the console.    We have experimented with a couple of different function overrides to catch this text, but don’t have a solution that works inside of SkyWeather that doesn’t have bad side effects.

If you do solve this problem, please let us know!

 

127.0.0.1 - - [15/Jul/2019 09:53:56] "GET /stream.mjpg HTTP/1.1" 200 -
WARNING:root:Removed streaming client ('127.0.0.1', 33240): [Errno 104] Connection reset by peer
Traceback (most recent call last):
  File "/usr/lib/python2.7/SocketServer.py", line 599, in process_request_thread
    self.finish_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 334, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python2.7/SocketServer.py", line 657, in __init__
    self.finish()
  File "/usr/lib/python2.7/SocketServer.py", line 716, in finish
    self.wfile.close()
  File "/usr/lib/python2.7/socket.py", line 283, in close
    self.flush()
  File "/usr/lib/python2.7/socket.py", line 307, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 32] Broken pipe

 

Acknowledgements

For the PiCamera basic streaming software:

https://picamera.readthedocs.io/en/release-1.13/recipes2.html

For the idea for using OpenCV for frame grabbing

https://www.pyimagesearch.com/2015/03/30/accessing-the-raspberry-pi-camera-with-opencv-and-python/