I get the news I need on the weather report

In which we publish a MJPEG stream from the BeagleBone Black.

Continuing with the series of post on OpenCV, webcams, and MJPEG, today we will look at streaming an MJPEG capture from the BBB.  Before I get into it though, you should know that I did try FFMPEG/avconv and VLC to stream video from the BBB using rtp, but the several seconds of latency made it unsuitable for my needs.  You should also know that I do not claim that this is the one true way to stream video from the BBB.

The libraries used:

  • ZeroMQ[1] and CZMQ[2] - used to create pub/sub connections between the BBB and software running on a desktop
  • OpenCV[3] - used to display the MJPEG stream

 

The subscriber was compiled and tested under Windows 7 using Visual Studio 2012; however, the code should compile under Linux with very few, if any, modifications.

Backgound:

I am working on a project that requires a video stream from the BBB be consumed in N places where N is a minimum of 2.  The stream will be processed using OpenCV, and because of the nature of this project, I need as little latency in the video stream as possible.

Theory of Operation:

The BBB will capture frames in MJPEG format from a webcam via a modified version of framegrabber.  The modified version of framegrabber can run indefinitely and outputs the frames as a series of ZeroMQ messages over a publish socket.  The clients will subscribe to the publish socket on the BBB using ZeroMQ and load each frame received into OpenCV.

The ZeroMQ pub/sub configuration allows many clients to connect to the published stream.  No synchronization is used between the publisher and subscriber; the the stream is treated as continuous, and the subscribers are free to connect and disconnect at will.

Results:

Single subscriber 640x480 - cpu use on the BBB ~4.3% and memory use ~0.8%

Multiple subscribers 640x480 - cpu use on BBB ~6.6% and memory use ~0.8%

Single subscriber 1920x1080 - cpu use on BBB ~23.2% and memory use ~3.5%

Using this setup, I have been able to stream frames with a resolution as high as 1920x1080 with little to no latency, but there is a limitation, the network.  When using this over Wifi with high resolutions or several clients running on one machine, I noticed the frame rate would drop the further I went from the router.  If you watch the output of the top command on the BBB as you get further from the router, you will see framegrabber's memory use begin to climb.  This is due to the publish socket buffering the data.  As you walk back towards the router, you will see the memory use drop until it, and the frame rate, stabilizes.  During this stabilization period you will probably experience delayed video that is displayed at a higher frame rate than normal as the buffer is flushed.

There are several things you can do reduce or eliminate this latency.

  1. If possible, use a wired connection
  2. Use an 802.11 N router and clients
  3. Make sure your WiFi router is optimally located
  4. Adjust the QoS settings of your router to give higher or highest priority to the traffic on the port you publish over

 

To reduce the amount of time it takes for subscribers to catch up once their connection has improved, the high water mark on the socket can be reduced.  This has the effect of dropping frames once too many are buffered and essentially reduces the amount of buffered data a subscriber has to process to get in sync.

The reader may find it interesting that it does not matter if the BBB publisher or the subscribers are started up first.  The publisher will simply dump data until at least one subscriber connects, and the subscribers will wait on the publisher.  In addition, you can kill the publisher while subscribers are connected, restart it with new (or the same) settings, and the subscribers will continue on.  The reader should verify this by changing the resolution after the subscriber or subscribers have connected.

Code:

framegrabberPub.c (17.96 kb) Publisher - you will need zhelpers.h

compile with

gcc framegrabberPub.c -lzmq -o framegrabberPub 

framegrabberSub.c (3.79 kb) C client

[BONUS]
framegrabberSub.py (2.52 kb) Python client

The Python client will display the stream with little latency until garbage collection occurs.  When this happens, the display will freeze and the buffered data on the BBB will increase.  Once garbage collection completes, the display will eventually synchronize much like the WiFi issue detailed above.

[UPDATE]
If your wireless router is capable of broadcasting at both 2.4 and 5 Ghz at the same time, you can improve performance when using a WiFi connection for both the publisher and the subscriber by having one connect at 2.4 Ghz and the other connect at 5 Ghz.

[UPDATE]
Added a link to zhelpers.h needed to compile the publisher.

[1] http://zeromq.org/
[2] http://czmq.zeromq.org/
[3] http://opencv.org/

Comments (6) -

  • Eric

    10/1/2013 5:59:30 AM | Reply

    great article,
    but I was wondering if I try to run
    gcc framegrabberPub.c -lzmq -o framegrabberPub
    I am getting :
    framegrabberPub.c:58:22: fatal error: zhelpers.h: No such file or directory
    compilation terminated.

    So I was wondering are there any dependancies, packages or any more files needed to run the Publisher's code ?

    I am trying to set up a MJPEG stream from the BBB which then is picked up by any other device (might be a PC, or a tablet, or a smartphone). The most important things are to keep latency to a minimum and anything with a decent resolution (800x600 upwards) and a decent framerate (15 fps upwards)...
    I was wondering though where could I get the unmodified framegrabber version ? I don't need the OpenCV and ZeroMQ combo ?!

  • K3vin

    10/9/2013 3:32:16 AM | Reply

    Hi!
    How do I get the zmq Lib for the BBB?
    Its not Available on Angstrom Packages :/ "xx" dosn't fix the problem it has to be installed to the main libs,

  • K3vin

    10/9/2013 4:24:02 AM | Reply

    Hi!
    So, I got ZeroMQ and CZMQ from their Respective Websites. Did ./configure and make install for both on the BBB.
    the
    "gcc framegrabberPub.c -lzmq -o framegrabberPub"
    ran without Errors.
    when i run it ./framegrabberPub
    it gives an Error saying
    "error while loading shared libraries: libzmq.so.3: cannot open shared object file: No such file or directory
    "

    Am I MIssing Something?

    • Lemoneer

      10/9/2013 9:03:48 AM | Reply

      Did you run
          sudo ldconfig
      after building to update the library cache?

  • DougP

    10/10/2013 5:40:04 AM | Reply

    Lemoneer, excellent stuff - thanks for posting!  I'm planning a project to read, display (to LCD) and record (to disk) a video stream from a camera like the Logitech C920, and it looks like you've done almost all of that on the BBB in one of your other posts (with the exception, perhaps, of saving to video file for later replay).

    Do you know where the several seconds of latency came from in this project?  Is it from FFMPEG / avconv, or from the transfer stream?  If it's the transfer rtp stream, then that won't concern me, but if you think FFMPEG / avconv is doing it, that would be a big problem for me, especially when playing the incoming stream to the display.

Loading