Tutorial – Python MJPEG Video Streaming on Raspberry Pi with 
Overlays
This article talks about how to stream video in MJPEG/MJPG format from your Raspberry Pi using Pure Python and openCV. It supports a frame rate > 24FPS and allows overlays and information to be added on a frame by frame basis.
This software is used the the new SkyWeather system to stream live video to the WeatherSTEM cloud for use by the public.
SkyWeather allows you to build your own weather station with a Sky Cam to take pictures of your Sky and display them the cloud on WeatherSTEM. You might even make it to the WeatherChannel!
All up-to-date information for SkyWeather is here.
What is MJPEG
Motion JPEG (M-JPEG or MJPEG) is a video compression format in which each video frame or interlaced field of a digital video sequence is compressed separately as an individual JPEG image. Originally developed for multimedia PC applications, M-JPEG is now used by video-capture devices such as digital cameras, IP cameras, and webcams, as well as by non-linear video editing systems. It is natively supported by the QuickTime Player, the PlayStation console, and web browsers such as Safari, Google Chrome, Mozilla Firefox and Microsoft Edge.
This format works really well for applications that want to be able to easily
process each frame on the fly, versus a compressed format like mp4 or mpeg which needs to be decompressed on the fly to be manipulated by the Raspberry Pi.
MJPEG Video Streaming in Pure Python
This software gives an example of how to stream MJPEG/MJPG video on the Raspberry Pi using the PiCamera library using overlays. This is the same software that is being used in the SwitchDoc Labs SkyWeather product in conjunction with WeatherSTEM.
It is designed to do Python video streaming in thread on SkyWeather, with single picture capability and the ability to add overlays / information on the fly.
This software requires a Rapsberry Pi 3B+ or greater.
Software Contents
streamtest.py
This Python program streams video using the openCV and the PiCamera library. Each frame is intercepted and various overlays and information is placed on the frame then sent to the video stream.
Defaults to streaming on:
http://localhost:443/
cvgrab.py
Grabs a single frame from the MJPEG video stream using openCV. Writes to a file “test.jpg”
GitHub Repository
You can find this software here:
https://github.com/switchdoclabs/SDL_Pi_MJPEGStream
Software Description
The streamtest.py software is pretty straight forward. There are three main sections to the streamtest.py software.
Streaming Handler
class StreamingHandler(BaseHTTPRequestHandler):
def do_GET(self):
if self.path == '/':
self.send_response(301)
self.send_header('Location', '/index.html')
self.end_headers()
elif self.path == '/index.html':
content = PAGE.encode('utf-8')
self.send_response(200)
self.send_header('Content-Type', 'text/html')
self.send_header('Content-Length', len(content))
self.end_headers()
self.wfile.write(content)
elif self.path == '/stream.mjpg':
self.send_response(200)
self.send_header('Age', 0)
self.send_header('Cache-Control', 'no-cache, private')
self.send_header('Pragma', 'no-cache')
self.send_header('Content-Type', 'multipart/x-mixed-replace; boundary=FRAME')
self.end_headers()
try:
while True:
with output.condition:
output.condition.wait()
frame = output.frame
# now add timestamp to jpeg
# Convert to PIL Image
cv2.CV_LOAD_IMAGE_COLOR = 1 # set flag to 1 to give colour image
npframe = np.fromstring(frame, dtype=np.uint8)
pil_frame = cv2.imdecode(npframe,cv2.CV_LOAD_IMAGE_COLOR)
#pil_frame = cv2.imdecode(frame,-1)
cv2_im_rgb = cv2.cvtColor(pil_frame, cv2.COLOR_BGR2RGB)
pil_im = Image.fromarray(cv2_im_rgb)
draw = ImageDraw.Draw(pil_im)
# Choose a font
font = ImageFont.truetype("/usr/share/fonts/truetype/freefont/FreeSans.ttf", 25)
myText = "SkyWeather "+dt.datetime.now().strftime('%Y-%m-%d %H:%M:%S')
# Draw the text
color = 'rgb(255,255,255)'
#draw.text((0, 0), myText,fill = color, font=font)
# get text size
text_size = font.getsize(myText)
# set button size + 10px margins
button_size = (text_size[0]+20, text_size[1]+10)
# create image with correct size and black background
button_img = Image.new('RGBA', button_size, "black")
#button_img.putalpha(128)
# put text on button with 10px margins
button_draw = ImageDraw.Draw(button_img)
button_draw.text((10, 5), myText, fill = color, font=font)
# put button on source image in position (0, 0)
pil_im.paste(button_img, (0, 0))
bg_w, bg_h = pil_im.size
# WeatherSTEM logo in lower left
size = 64
WSLimg = Image.open("WeatherSTEMLogoSkyBackground.png")
WSLimg.thumbnail((size,size),Image.ANTIALIAS)
pil_im.paste(WSLimg, (0, bg_h-size))
# SkyWeather log in lower right
SWLimg = Image.open("SkyWeatherLogoSymbol.png")
SWLimg.thumbnail((size,size),Image.ANTIALIAS)
pil_im.paste(SWLimg, (bg_w-size, bg_h-size))
# Save the image
buf= StringIO.StringIO()
pil_im.save(buf, format= 'JPEG')
frame = buf.getvalue()
self.wfile.write(b'--FRAME\r\n')
self.send_header('Content-Type', 'image/jpeg')
self.send_header('Content-Length', len(frame))
self.end_headers()
self.wfile.write(frame)
self.wfile.write(b'\r\n')
except Exception as e:
traceback.print_exc()
logging.warning(
'Removed streaming client %s: %s',
self.client_address, str(e))
else:
self.send_error(404)
self.end_headers()
The function has three sections in the mpeg stream. First you manipulate the jpeg file from the mjpeg stream to a numpy image and then to a PIL image to work on.
frame = output.frame
# now add timestamp to jpeg
# Convert to PIL Image
cv2.CV_LOAD_IMAGE_COLOR = 1 # set flag to 1 to give colour image
npframe = np.fromstring(frame, dtype=np.uint8)
pil_frame = cv2.imdecode(npframe,cv2.CV_LOAD_IMAGE_COLOR)
#pil_frame = cv2.imdecode(frame,-1)
cv2_im_rgb = cv2.cvtColor(pil_frame, cv2.COLOR_BGR2RGB)
pil_im = Image.fromarray(cv2_im_rgb)
Next you manipulate the PIL image by adding information, overlays, images, etc.
Then finally, you convert it back to a jpeg file and send it off to the video stream. The resulting image looks like this in SkyWeather:
This gives us great flexibility to add overlays and more data as the situation requires.
Streaming Service
class StreamingServer(SocketServer.ThreadingMixIn, HTTPServer):
allow_reuse_address = True
daemon_threads = True
#with picamera.PiCamera(resolution='640x480', framerate=24) as camera:
#with picamera.PiCamera(resolution='1920x1080', framerate=24) as camera:
with picamera.PiCamera(resolution='1296x730', framerate=24) as camera:
output = StreamingOutput()
camera.start_recording(output, format='mjpeg')
camera.annotate_foreground = picamera.Color(y=0.2,u=0, v=0)
camera.annotate_background = picamera.Color(y=0.8, u=0, v=0)
try:
address = ('', 443)
server = StreamingServer(address, StreamingHandler)
server.serve_forever()
finally:
camera.stop_recording()
Frame Grabbing Software
The cvgrab.py software used to grab individual frames is very short:
import cv2
print "starting grab"
cap = cv2.VideoCapture('http://localhost:443/stream.mjpg')
ret, frame = cap.read()
print "found frame"
#cv2.imshow('Video', frame)
cv2.imwrite("test.jpg",frame)
print "done"
cap.release()
print "after release"
Known Issues
We have been running this software for weeks with no significant issues. One issue that we have run into with the cvgrab software is that the streaming software throws a handled exception each time a frame is grabbed, which while it doesn’t kill the streamer, it prints unwanted data printed to the console. We have experimented with a couple of different function overrides to catch this text, but don’t have a solution that works inside of SkyWeather that doesn’t have bad side effects.
If you do solve this problem, please let us know!
127.0.0.1 - - [15/Jul/2019 09:53:56] "GET /stream.mjpg HTTP/1.1" 200 -
WARNING:root:Removed streaming client ('127.0.0.1', 33240): [Errno 104] Connection reset by peer
Traceback (most recent call last):
File "/usr/lib/python2.7/SocketServer.py", line 599, in process_request_thread
self.finish_request(request, client_address)
File "/usr/lib/python2.7/SocketServer.py", line 334, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python2.7/SocketServer.py", line 657, in __init__
self.finish()
File "/usr/lib/python2.7/SocketServer.py", line 716, in finish
self.wfile.close()
File "/usr/lib/python2.7/socket.py", line 283, in close
self.flush()
File "/usr/lib/python2.7/socket.py", line 307, in flush
self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 32] Broken pipe
Acknowledgements
For the PiCamera basic streaming software:
https://picamera.readthedocs.io/en/release-1.13/recipes2.html
For the idea for using OpenCV for frame grabbing
https://www.pyimagesearch.com/2015/03/30/accessing-the-raspberry-pi-camera-with-opencv-and-python/
