Lemon Difficult

In which we see that not everyone understands what it is to be unique.

I am currently wrapped up in an EDI implementation, and while I take issue with a great many things in the EDI world, this post will be about just one of the many flaws in a piece of software used to help manage an EDI workflow.

Setting the stage.

Let us talk in somewhat generic, higher level terms so that we do not get bogged down in the complexities of EDI TSETS (transmissions).

Suppose in our database we have a simple parent/child relationship.  For a given parent, you may have one or more children.  Now let us also suppose each child has a unique identifier used to link the child record to records related to it so that we have something like this:

On the EDI side, suppose the TSET has some ‘header’ data that represents the Parent, that there is a group of segments we can loop over that represent the children, and that some of the data for a child is optional.  When the optional child data exists, we would like to insert it into the ChildData table.

Thus far everything seems pretty straight forward and easy.

Enter the EDI software.

As it turns out, the EDI software does not allow you to insert a record and retrieve the Id assigned by the database (at least there is neither an obvious way nor one that does not involve the occult).  Not to worry, the EDI software has a way generate unique ids that we can use when we insert child records as well as child data records.  This is accomplished by an operator called “Replace With Unique String”.

So we wire everything up, load a test TSET, and tell the software to import it.  Tada!  Several of the children have the same Id.  At this point, frustrated with how soul crushingly difficult everything thing is with this software, you sigh because you know you have been let down again.

Back in the interface for building the map, you dig up the Help Info for the “Replace With Unique String” operator and find the following:

Oh lovely you think; It generates a unique 12 character strings.  Twelve seems a bit short to be unique in space and time, but hey we paid good money for this software.  We are sure they know what they are doing.  And it is guaranteed to be unique even if called from separate instances in the same millisecond!

Yes.  Yes they did.  The developers who implemented this software used a timestamp encoded in some special string format as a unique identifier.

And you sigh again because you know the poor developers must have been forced to work on ancient computers where millisecond accuracy is good enough.  Their hardware was not fast enough to loop over items any quicker.  Alas, our server is.  It would seem our server can process several records in a millisecond as is evident by the fact that many records share the same id.  Even more tragic, sometimes if you click the import button at just the right time with the right load on the server, they all get unique ids.

For anyone out there thinking they will solve their unique Id requirement with a timestamp, please stop.  Processors are fast, your computer is fast, many things may happen during the time span measurable by the resolution of your typical Date.getTime() call (or your language’s equivalent).

In which we enable WiFi on the Wandboard Dual and Quad.

I have come to the time in a project where my glorious BeagleBone Black is sturggling to keep up.  What to do?  Fire up the Wandboard Quad!  As with the BeagleBone Black, I prefer to run Arch Linux Arm. And as before, WiFi does not immediately work (despite WiFi being built into the Wandboard Quad).  Luckily getting WiFi running is much easier than on the BeagleBone Black.

The WiFi chip on the Wandboard Dual and Quad is a Broadcom BCM4329 connected via SDIO and requires both a firmware and nvram in order to work.

First, we will load the nvram.  The command below will fetch the nvram from Freescale's github (Freescale is the maker of the processor used on the Wandboard) and place it in the proper directory.

wget -c https://raw.github.com/Freescale/meta-fsl-arm-extra/master/recipes-bsp/broadcom-nvram-config/files/wandboard/nvram.txt
sudo mv -v nvram.txt /lib/firmware/brcm/brcmfmac-sdio.txt

Next we need to rename the firmware already present in Arch Linux Arm.  This is required because the kernel Arch uses for the Wanboard is v3.0.35-3 and for kernels older than v3.13, the SDIO driver used generic firmware names^[2].

cp -v /lib/firmware/brcm/brcmfmac4329-sdio.bin /lib/firmware/brcm/brcmfmac-sdio.bin

Now reboot the Wandboard.  When it comes back up, the output of ip link should list wlan0 as an option.

With the WiFi adapter recognized, we can connect to the router.

Install WPA

pacman -S wpa_actiond

Create a base config file

cp -v /etc/netctl/examples/wireless-wpa-configsection /etc/netctl/wireless_wpa_configsection

Generate the required wpa_supplicant config data

wpa_passphrase SSID PASSWORD

Insert the config section into your config file

nano /etc/netctl/wireless_wpa_configsection

Now test out the connection

netctl start wireless_wpa_configsection

Assuming no errors, set WiFi to load at boot

systemctl enable netctl-auto@wlan0.service

Note 1: Loading the firmware and nvram is not necessarily specific to Arch Linux Arm. If your flavor of Linux does not recognize the WiFi adapter, give it a go.

Note 2: Since the adapter is connected using SDIO, it will not show up in the output of lspci -k or lsusb -v

[1] Wandboard
[2] Broadcom Linux drivers

In which we make OpenCV play nice with Anaconda under Windows and roll libjpeg-turbo in along the way.

You may have noticed, if you have been reading my blog, that I like to prototype in Python.  I find Python to be almost as fun as Lisp when it comes to building up a solution without knowing exactly what the solution is going to look like.  The reason I typically start in Python is the wealth of libraries; however, managing a Python install can become a real pain.  Thanks to the people over at Continuum Analytics, you can be spared most of it.

Continuum Analytics produces a fantastic distribution known as Anaconda (or Miniconda depending on the route you choose).  I won't go into all the reasons I think you should give it a try, but if you program in Python, you should check them out.  The only shortcoming I have come across is the Windows build does not include OpenCV, but we will remedy that today.  Along the way, I will also show you how to build OpenCV against libjpeg-turbo for better jpeg performance.

Things you will need:

1. CMake: Used to generate makefiles/solutions to build OpenCV.
2. Visual Studio: This article is geared towards VS, but you should be able to follow roughly the same steps for other compilers.
3. OpenCV: The reason for this excursion.  I downloaded the self extracting Windows package.  You can just grab the source if you like.
4. libjpeg-turbo: An optimized crossplatform library for working with jpegs.
5. Anaconda (or Miniconda): My (and soon to be yours) favorite Python distribution.

Caution:

Make sure your versions match, i.e. if you install 64 bit Anaconda, install the 64 bit version of libjpeg-turbo and build OpenCV for 64 bits.

If you use Miniconda to make your default Python installation Python 3 (as I have done) you will need to make a Python 2 environment.  Please see here.

 To the Bat Cave!

1. Run the self extracting OpenCV archive if necessary, and extract OpenCV to a place you will remember, e.g. C:\OpenCV
2. Run the installer for libjpeg-turbo
3. If your default Python install is Python 3
1. Right click on the cmake/bin directory while holding down Shift
2. Click "Open command window here"
3. Activate your Python 2 environment with

   activate ENV_NAME

4. Launch cmake-gui (from the command prompt in step 3 if necessary)
5. Set "Where is the source code:" to the path in step 1
6. Set "Where to build the binaries:" to where you normally build libs, e.g. C:\Libs\OpenCV
7. Click Configure and wait for it to complete
8. Check the Advanced checkbox
9. Locate the option BUILD_JPEG and uncheck it
10. Click the Add Entry button and add
    • Name: JPEG_LIBRARY
    • Type: FILEPATH
    • Value: path to the static libjpeg-turbo64 e.g. C:/libjpeg-turbo64/lib/jpeg-static.lib
11. Click the Add Entry button and add
    • Name: JPEG_INCLUDE_DIR
    • Type: PATH
    • Value: path to the libjpeg-turbo64 include e.g. C:/libjpeg-turbo64/include
12. If you see a message in the Configure output that says PYTHON_INCLUDE_DIR and/or PYTHON_LIBRARY are not found, you will need to set them by hand by clicking in the Value column and entering the path. On my system (Windows 8.1 Pro) these are C:/Users/lemoneer/Miniconda3/envs/python-2/include and C:/Users/lemoneer/Miniconda3/envs/python-2/libs/python27.lib respectively.
13. Click Configure again and wait for it to complete
14. Verify all is well in the output of Configure
15. Click Generate
16. Open the generated solution file in your build directory from step 6
17. Change the build to Release
18. Build the Solution
19. Expand CMakeTargets in the Solution Explorer
20. Right click on INSTALL
21. Click Build
22. Once the build is complete, update your path to include the OpenCV dlls
23. You may Clean the Build at the solution level to remove intermediate files you do not need (this wont affect the install).

It is worth nothing that there are other build options worth considering like building the examples.  Enable/disable to your heart's content then click Configure followed by Generate and Build.

If all goes well, you will have OpenCV dlls you can use from C/C++ as well as from your Python 2 environment.

In which we assemble a portable, Emacs based LISP development environment under Windows.  This being good for the soul.

For Carlos, happy birthday.  You will come to understand... they all do.
For Josh, you were right about Lisp.

Before we begin, let me say if you are new to Lisp, you should check out Quicklisp.  Quicklisp provides a convenient way to setup and maintain a Lisp development environment with a large set of libraries.  However, if you would like a portable Lisp development environment and/or would like to have more control over (or a better understanding of) your Emacs based setup, please read on.

What you will need:

1. The latest version of Emacs for Windows.  Download the zip.
2. The latest CVS snapshot of SLIME.  Download the tar archive.
3. One or more of the following Lisp implementations:

• CCL: Checkout the latest release version from SVN.

• CLISP

• ECL: You will have to compile from source.

Optional:

• Par Edit: helps **keep parentheses balanced**

And we are off!

1. Extract Emacs to a location of your choice, e.g. emacs-24.3 (tip include the version in the directory names so you can tell which version you are running.
2. Create a directory named 'home' in the root of your emacs directory, e.g. emacs-24.3/home
3. Create a directory named 'bin' in the root of 'home', e.g. emacs-24.3/home/bin
4. Deploy Lisp implementation(s)
• CLISP
• Extract to a directory in bin, e.g. emacs-24.3/home/bin/clisp-2.49
• ECL
• Compile from source and install to a directory in bin, e.g. emacs-24.3/home/bin/ecl-13.5.1
• CCL
• Checkout from SVN to a directory in bin, e.g. emacs-24.3/home/bin/ccl
• Tip: keep CCL up to date between version releases with SVN's update command.
• Modify ccl/level-1/linux-files.lisp line 939 #+windows-target to set HOME from the 'HOME' environment variable.  This allows us to control where CCL writes files reducing debris and making it portable.

  #+windows-target
  ;; Matthew Witherwax (lemoneer) 01MAY2013
  ;; Modified the Windows target to use the value of the HOME environment
  ;; variable if it exists; otherwise, it falls back to the OS user database.
  ;; This allows CCL to be run portably under Windows without leaving debris
  ;; behind.
  (or
   (getenv "HOME")
   (dolist (k '(#||"HOME"||# "USERPROFILE"))
     (with-native-utf-16-cstrs ((key k))
       (let* ((p (#__wgetenv key)))
     (unless (%null-ptr-p p)
       (return (get-foreign-namestring p)))))))
  

5. Create the file site-start.el in site-lisp in the root of your Emacs directory, e.g. emacs-24.3/, and insert the code below. This allows us to set the ‘HOME’ environment variable when we launch Emacs.

   ; remove etc from the path
       (defvar %~dp0
           (substring data-directory 0
           (- (length data-directory) 4)))
       ; create appened home/
       (defvar home-dir (concat %~dp0 "home/"))
       (setenv "HOME" home-dir)
   

6. Extract the latest slime csv snapshot to site-lisp, e.g. emacs-24.3/site-lisp/slime-2013-12-12
7. Optional: Copy paredit to /lisp in the root of your Emacs directory, e.g. emacs-24.3/lisp
8. Launch Emacs by double clicking runemacs.exe in the bin directory of the Emacs directory, e.g. emacs-24.3/bin, and type C-x C-f ~/.emacs to open the Emacs config file.   Insert the following:

   (set-language-environment "utf-8")
   
   ;;; Add slime
   (add-to-list 'load-path "~/site-lisp/slime-2013-12-12")  ;or wherever you put it
   
   ;;; CCL Note that if you save a heap image, the character
   ;;; encoding specified on the command line will be preserved,
   ;;; and you won't have to specify the -K utf-8 any more.
   
   ;; SLIME and CLISP will complain about a missing temp directory without this
   (setq temporary-file-directory (expand-file-name "~/temp"))
   
   ;; Path to clisp full so we can build the call
   (setq path-clisp (expand-file-name "~/bin/clisp-2.49/full/"))
   
   ;;; Configure available Lisp
   ;;; Leave out the one(s) you did not install
   (setq slime-lisp-implementations
   	`(
   		(ccl (,(expand-file-name "~/bin/ccl/wx86cl64.exe") "-K utf-8"))
   		(clisp (,(replace-regexp-in-string "@" path-clisp "@lisp.exe") 
   			,(replace-regexp-in-string "@" path-clisp "-B@") 
   			,(replace-regexp-in-string "@" path-clisp "-M@lispinit.mem")
   			"-ansi"))
   		(ecl (,(expand-file-name "~/bin/ecl-13.5.1/ecl.exe")))))
   
   ;;; setup slime
   (require 'slime)
   (setq slime-net-coding-system 'utf-8-unix)
   (slime-setup '(slime-fancy))
   
   ;;; setup paredit if you installed it
   (autoload 'paredit-mode "paredit"
     "Minor mode for pseudo-structurally editing Lisp code." t)
   (add-hook 'emacs-lisp-mode-hook       (lambda () (paredit-mode +1)))
   (add-hook 'lisp-mode-hook             (lambda () (paredit-mode +1)))
   (add-hook 'lisp-interaction-mode-hook (lambda () (paredit-mode +1)))
   (add-hook 'scheme-mode-hook           (lambda () (paredit-mode +1)))
   
   ;;; This is disabled to stop paredit from running in the REPL
   ;;; Feel free to uncomment this to turn it back on
   
   ;;;; (add-hook 'slime-repl-mode-hook (lambda () (paredit-mode +1)))
   ;;;; ;; Stop SLIME's REPL from grabbing DEL,
   ;;;; ;; which is annoying when backspacing over a '('
   ;;;; (defun override-slime-repl-bindings-with-paredit ()
   ;;;; (define-key slime-repl-mode-map
   ;;;; 	(read-kbd-macro paredit-backward-delete-key) nil))
   ;;;; (add-hook 'slime-repl-mode-hook 'override-slime-repl-bindings-with-paredit)
   
   ;;; bonus split the screen in to two side by side windows
   (defun 2-windows-vertical-to-horizontal ()
     (if (and
          (= (length (window-list)) 2) ; only split if there are 2 windows
          (window-combined-p)) ; and they are vertical
         (progn
   	(delete-window)
   	(split-window-horizontally))))
   
   (add-hook 'emacs-startup-hook '2-windows-vertical-to-horizontal)
   
   (when (display-graphic-p)
     (add-to-list 'default-frame-alist '(left . 0))
     (add-to-list 'default-frame-alist '(top . 0))
   )
   
   (w32-send-sys-command 61488)
   

9. Create a batch file called runslime.bat and insert the following command.  This will allow us to launch Emacs with slime and your default Lisp running.

   @ECHO OFF
   C:\emacs-24.3\bin\runemacs.exe --eval=(slime)

 If you followed all the steps, when you run runslime.bat, you should see something like this

Take this REPL, brother, and may it serve you well.

In which we use OpenCV to track a laser.

Building on my article about using HSV thresholding to isolate features in video frames, below is a video showing the use of OpenCV to track a red laser.  The majority of the algorithm is the same as presented in the previous article; the only addition is actually determining the area of interest after thresholding.  In order to preventing spoiling anyone's fun, I am not going to post the code as of now.

Hint: think about how to find the largest area in white after thresholding.

If you would like to compare algorithms or just want to know what I did, drop me a note via the contact form.

In preliminary testing, the algorithm seems robust enough to handle changing lighting conditions as well as differing background colors; however, the angle of reflection of the laser can cause issues.  For best results, the laser should be relatively perpendicular to the image plane.  As the laser approaches being parallel to the image plane, less laser light is reflected back to the camera.  The same applies to irregular objects or surfaces that scatter the laser at angles away from the camera.  This can be seen at the end of the video when the laser is reflected off the lamp shade.

Testing was done using a Logitech C920.  Depending on the laser used and ambient lighting conditions, turning off auto white balance may prevent the intensity of the laser from being interpreted as white.

As far as applications, I leave that to the reader.

In which we join files line by line.

Working in IT, there is no shortage of "grunt work" that is amenable to one off scripts or applications.  In an effort to prevent others (and possibly myself) from having to reimplement these, I am publishing these as useLESS tools.

useLESS tools: scripts or programs written to paper over the shortcomings or deficiencies in applications or processes.  You could use less, and they would be made useless, if you could reimplement everything in light of the experience gained by doing it the last time.

Today I give you our first useLESS tool - fjoin

At work I am currently involved in integrating with an external vendor.  As part of the process, I have been provided with pseudo realistic test data to process.  Being pseudo realistic, it doesn't really match with what our system expects and must be "massaged".  That is to say, nothing works.

In particular, I have two text files.  One text file has the IDs used by the vendor, and the other has the IDs our system expects.  I need a way to turn these into SQL update statements, and it needs to be repeatable so I can rerun it as testing progresses.  What to do?

Using notepad++ or you favorite editor, it is fairly easy to record a macro that inserts

UPDATE [TABLE] SET [FIELD] = '

before each line of one file and

' WHERE ID = 

at the beginning of each line in the other so that if you took line x from the first file and line x from the second file and joined them, you would get something like

UPDATE [TABLE] SET [FIELD] = 'OURID_123' WHERE ID = 42

The real trick is merging the two files line by line.  fjoin does just this.  It takes two or more files and joins them line by line stopping at the end of the shortest file.  The result is sent to the terminal so you can pipe this into another program or redirect to a file.

If you would like to create an executable from this code, you can use pyinstaller

pysinstaller -F fjoin.py

This will create a single (rather large) executable for you.

import sys

if len(sys.argv) == 1:
    print 'No files specified'
    sys.exit(0)

files = []
# skip the name of the program
for arg in sys.argv[1:]:
    try:
        files.append(open(arg))
    except IOError:
        print 'ERROR:', arg, 'does not exist'
        sys.exit(1)

reading = True
composite = []

while reading:
    for f in files:
        line = f.readline()
        # stop as soon as a file comes up short
        if not line:
            reading = False
            break
        else:
            # remove newline
            composite.append(line.rstrip('\n'))
    # only output if we joined across all files
    # in other words, no partial lines
    if len(composite) == len(files):
        print ''.join(composite)
    composite = []

for f in files:
    f.close()

fjoin.py (2.36 kb)

In which we optimize OpenCV on the BeagleBone Black.

If you have been paying attention to the BBB group on Google Groups, you may have discovered a lively thread on webcams [1]. As part of this thread, I have been working with Michael Darling to realize the best performance possible when using OpenCV to process an MJPG stream from a webcam on the BBB.  OpenCV relies on libjpeg when loading the MJPG frames to OpenCV Mats, and libjpeg is not the fastest of jpeg libraries.  However, there is another option - libjpeg-turbo [2].

While it is technically possible to compile OpenCV with libjpeg-turbo support on the BBB, you will have fewer issues and spend less time compiling if you use a cross compiler on a more powerful computer.  Michael has written a guide to cross compiling OpenCV.  Below you will find the guide as a webpage for online viewing or a pdf for download as well as the latest code to capture frames from a webcam and convert them to OpenCV Mats.  The guide is currently a draft, and we would appreciate any feedback you can provide.  Many thanks to Michael for taking the time to write this up.

Note: The code in the guide and the latest code are slightly different.

1. -o is used to indicate which frames to convert to OpenCV Mats and requires an integer argument.
       -o 1 will convert every frame
       -o 2 will convert every 2nd (ever other) frame, and so on.
       The default is 1.
2. -p is similar to -o in the original framegrabber.  However, it doesn't actually output anything.  It just controls if any frames are to be converted.
3. Captured count and processed count variables have been renamed and moved to the top.
4. Formatting has been corrected.
5. Setting of the Mat size now uses the width and height variables.

The guide will be updated to reflect these changes in a future release, but in the meantime, you will need to adjust the command line arguments specified in the guide, namely replace -o with -p.

Guide [DRAFT]
    BBB_30fps.html
    BBB_30fps.pdf (204.32 kb)
Code
    framegrabberCV.c (19.77 kb)

[1] problems with webcams
[2] libjpeg-turbo

In which we publish a MJPEG stream from the BeagleBone Black.

Continuing with the series of post on OpenCV, webcams, and MJPEG, today we will look at streaming an MJPEG capture from the BBB.  Before I get into it though, you should know that I did try FFMPEG/avconv and VLC to stream video from the BBB using rtp, but the several seconds of latency made it unsuitable for my needs.  You should also know that I do not claim that this is the one true way to stream video from the BBB.

The libraries used:

• ZeroMQ^[1] and CZMQ^[2] - used to create pub/sub connections between the BBB and software running on a desktop
• OpenCV^[3] - used to display the MJPEG stream

The subscriber was compiled and tested under Windows 7 using Visual Studio 2012; however, the code should compile under Linux with very few, if any, modifications.

Backgound:

I am working on a project that requires a video stream from the BBB be consumed in N places where N is a minimum of 2.  The stream will be processed using OpenCV, and because of the nature of this project, I need as little latency in the video stream as possible.

Theory of Operation:

The BBB will capture frames in MJPEG format from a webcam via a modified version of framegrabber.  The modified version of framegrabber can run indefinitely and outputs the frames as a series of ZeroMQ messages over a publish socket.  The clients will subscribe to the publish socket on the BBB using ZeroMQ and load each frame received into OpenCV.

The ZeroMQ pub/sub configuration allows many clients to connect to the published stream.  No synchronization is used between the publisher and subscriber; the the stream is treated as continuous, and the subscribers are free to connect and disconnect at will.

Results:

Single subscriber 640x480 - cpu use on the BBB ~4.3% and memory use ~0.8%

Multiple subscribers 640x480 - cpu use on BBB ~6.6% and memory use ~0.8%

Single subscriber 1920x1080 - cpu use on BBB ~23.2% and memory use ~3.5%

Using this setup, I have been able to stream frames with a resolution as high as 1920x1080 with little to no latency, but there is a limitation, the network.  When using this over Wifi with high resolutions or several clients running on one machine, I noticed the frame rate would drop the further I went from the router.  If you watch the output of the top command on the BBB as you get further from the router, you will see framegrabber's memory use begin to climb.  This is due to the publish socket buffering the data.  As you walk back towards the router, you will see the memory use drop until it, and the frame rate, stabilizes.  During this stabilization period you will probably experience delayed video that is displayed at a higher frame rate than normal as the buffer is flushed.

There are several things you can do reduce or eliminate this latency.

1. If possible, use a wired connection
2. Use an 802.11 N router and clients
3. Make sure your WiFi router is optimally located
4. Adjust the QoS settings of your router to give higher or highest priority to the traffic on the port you publish over

To reduce the amount of time it takes for subscribers to catch up once their connection has improved, the high water mark on the socket can be reduced.  This has the effect of dropping frames once too many are buffered and essentially reduces the amount of buffered data a subscriber has to process to get in sync.

The reader may find it interesting that it does not matter it the BBB publisher or the subscribers are started up first.  The publisher will simply dump data until at least one subscriber connects, and the subscribers will wait on the publisher.  In addition, you can kill the publisher while subscribers are connected, restart it with new (or the same) settings, and the subscribers will continue on.  The reader should verify this by changing the resolution after the subscriber or subscribers have connected.

Code:

framegrabberPub.c (17.96 kb) Publisher - you will need zhelpers.h

compile with

gcc framegrabberPub.c -lzmq -o framegrabberPub 

framegrabberSub.c (3.79 kb) C client

[BONUS]
framegrabberSub.py (2.52 kb) Python client

The Python client will display the stream with little latency until garbage collection occurs.  When this happens, the display will freeze and the buffered data on the BBB will increase.  Once garbage collection completes, the display will eventually synchronize much like the WiFi issue detailed above.

[UPDATE]
If your wireless router is capable of broadcasting at both 2.4 and 5 Ghz at the same time, you can improve performance when using a WiFi connection for both the publisher and the subscriber by having one connect at 2.4 Ghz and the other connect at 5 Ghz.

[UPDATE]
Added a link to zhelpers.h needed to compile the publisher.

[1] http://zeromq.org/
[2] http://czmq.zeromq.org/
[3] http://opencv.org/

In which we learn how to turn a buffer of bytes into an image OpenCV can work with.

If you followed my previous post, you may have realized you can produce, with very few modifications, a stream of MJPEG (jpeg) images captured from the webcam. You may also be left wondering how to work with these images using OpenCV. It might actually be easier than you imagine.

The little bit of Python code below will display a single image. The code loads an image from a file into a buffer, converts the buffer into an OpenCV image, and displays the image. If you are receiving the image as a buffer already, you can forgo the loading step.

import cv2
from cv2 import cv
import numpy as np

# let's load a buffer from a jpeg file.
# this could come from the output of frameGrabber
with open("IMAGE.jpg", mode='rb') as file:
    fileContent = file.read()

# convert the binary buffer to a numpy array
# this is a requirement of the OpenCV Python binding
pic = np.fromstring(fileContent, np.int8)

# here is the real magic
# OpenCV can actually decode a wide variety of image formats
img = cv2.imdecode(pic, cv.CV_LOAD_IMAGE_COLOR)

cv2.namedWindow("image")
while True:
    cv2.imshow("image", img)
    
    key = cv2.waitKey(20)
    if key == 27: # exit on ESC        
        break
    
cv2.destroyAllWindows()

For those of you working in C, you can convert a buffer into an image with this (tested in Visual Studio 2012)

#include <stdio.h>
#include <malloc.h>
#include "opencv2\core\core_c.h"
#include "opencv2\highgui\highgui_c.h"

// used to load a jpeg image into a buffer
int load_buffer(const char *filename, char **result) 
{ 
	int size = 0;
	FILE *f = fopen(filename, "rb");
	if (f == NULL) 
	{ 
		*result = NULL;
		return -1; // -1 means file opening fail 
	} 
	fseek(f, 0, SEEK_END);
	size = ftell(f);
	fseek(f, 0, SEEK_SET);
	*result = (char *)malloc(size+1);
	if (size != fread(*result, sizeof(char), size, f)) 
	{ 
		free(*result);
		return -2; // -2 means file reading fail 
	} 
	fclose(f);
	(*result)[size] = 0;
	return size;
}

int main() 
{ 
	char *buffer; 
	int size;
	CvMat mat;
	IplImage *img;

	// load jpeg file  into buffer
	// this could come from the output of frameGrabber
	size =load_buffer("IMG.jpg", &buffer);
	if (size < 0) 
	{ 
		puts("Error loading file");
		return 1;
	}

	// create a cvMat from the buffer
	// note the params: height, width, and format
	mat = cvMat(1080, 1920, CV_8UC3, (void*)buffer);
	// magic sauce, decode the image
	img = cvDecodeImage(&mat, 1);

	// show the image
	cvShowImage("image", img );

	// wait for a key
	cvWaitKey(0);

	// release the image
	cvReleaseImage(&img);


	return 0;
}

For further information see the OpenCV Docs.

OpenCVjpeg.py (2.24 kb)
OpenCVjpeg.c (2.78 kb)

In which we discover the limits of webcams connected to the BeagleBone Black.

As the previous couple of posts may have hinted, I am currently working on a computer vision application.  On the hardware side, I am using a webcam connected to a BeagleBone Black to capture and process images.  Finding the right camera and software configuration seems to be a challenge many people are trying to overcome.  The following is what I have learned through experimentation.

During my first foray into the world of webcams on the BBB, I chose the PS3Eye.  The PS3Eye has been used for many computer vision applications thanks to its ability to produce uncompressed 640x480 images at up to 60 FPS or uncompressed 320x240 at up to 120 FPS.  The ability to capture uncompressed images at high frame rates plus being available for $16.98 would normally make the PS3Eye a fantastic choice; however, we are dealing with the BBB.

If you plug a PS3Eye into the BBB and fire up an OpenCV application to capture at 640x480, you will receive "Select Timeout" errors instead of a video stream.  If you do the same but with the resolution set to 320x240, it will work.  It turns out the PS3Eye transfers data in bulk mode over USB.  In bulk mode, you are guaranteed to receive all of the transmission; however, you are not guaranteed timing.  What is essentially happening is the PS3Eye is saturating the bulk allotment on the USB.  The reason you encounter this problem at 640x480 and not 320x240 is because OpenCV with Python sets the frame rate to 30 FPS and provides no way to change it.  We can calculate the amount of data put on the bus as follows:

Height * Width * (Channels * 8) * FPS

So for our uncompressed images at 640x480 we have:

640 * 480 * (3 * 8) * 30 = 221184000 bits/s or ~26.36 MB/s

and 320x240 is ~6.59 MB/s

As OpenCV with Python does not allow you to set frame rate, I modified v4l2grab^[1] to accept frame rate as a command line argument.  With this, I discovered you can capture images from the PS3Eye at 640x480 as long as you set the frame rate to 15 FPS or less.  You can also capture images at 320x240 at up to 60 FPS.  The astute reader will notice that 640 * 480 * (8 * 3) * 15 = 320 * 240 * (8 * 3) * 60 which is ~13.2 MB/s.  In other words, the USB on the BBB taps aout at ~13.2 MB/s for bulk transfers.

At this point you might be thinking you do not have to worry about frames per second because you will only take still shots.  It turns out uvc under Linux does not support still image capture^[2].  In order to capture an image, you open the webcam the same way you would to capture a stream; however, you just grab one frame (or more if needed).

If you would like to capture 640x480 or larger images at 30 FPS or faster, all is not lost, but you will need a webcam that supports some sort of native compression.  In my case, I am using a Logitech C920.  It can compress using both H264 and MJPEG.  If you want to capture a video stream, H264 is probably your best choice as it should have fewer compression artifacts.  It you are after still shots, MJPEG will be your friend.

MJPEG typically compresses each frame as a separate jpeg^*.  Since MJPEG uses intra-frame compression, you only need to capture one image for a still shot.  H264 uses inter-frame compression - meaning it relies on information from several frames to determine how to compress the current frame.  In order to reconstruct the frame, you need all the ones involved in the compression.  I know the last two sentences are a great simplification, but they suffice for our discussion.

In order to test the different combinations of frame rates and encodings, I extended the v4l2 capture sample available from the project's website^[3].  To the base sample I added the ability to specify image dimensions, frame rate, and pixel format (ie compression).  I also added handling for CTRL-C so the webcam is not left in an inconsistent state if you kill the program with CTRL-C, and the ability to set the timeout value and maximum number of timeouts.

The program is available here framegrabber.c.

Please note this software is not finished.  I am publishing it now so others may use it to determine the capabilities of their webcams, but I will be improving and extending it in the future.  You may consider the capture timing functionality described in 1 below to be complete while the saving of frames described in 2 will change.

To compile framegrabber you must have the development files for v4l2 installed.

Compile with:

gcc framegrabber.c -o framegrabber

At this time, framegrabber is intended to be used in one of two ways.

1.  Timing frame capture rates.

To time frame capture rates, simply pass framegrabber to time, omit the -o switch, and set -c to the number of frames you would like to capture.  Omitting -o instructs the program to simply discard all captured frames.  In this mode, framegrabber will capture c number of frames from the webcam as fast as possible.

Here is the simplest case:

time ./framegrabber -c 1000

And here we set the pixel format, image dimensions, and frame rate:

time ./framegrabber -f mjpeg -H 480 -W 640 -c 1000 -I 30

Have a look at all the other command line switches to get a sense of the possibilities.

2.  Capturing frames from a webcam

As mentioned above, I extended the application v4l2grab to support setting the frame rate.  v4l2grab allows you to capture jpeg images from webcams that support the YUYV format.  It grabs frames in YUYV format and then converts the frames to jpeg.

When capturing frames with framegrabber, the raw frame is written out.  No conversion to jpeg is done.  This is mostly a proof on concept to show that frames captured in MJPEG format are individual jpegs and can be written out without further processing.  This has been tested with a Logitech C920, and the output is indeed an jpeg image.  Capturing in H264 and YUYV format will also work, but you will not be able to simply open the resulting file in your favorite image editor.

Currently there is no way to specify the filename for the frame or frames captured, and if -c is greater than one, the first c - 1 frames will be overwritten by frame_c.  To capture a frame, include the -o switch and set -c to one.  The resulting frame will be written to capture.jpg.

./framegrabber -f mjpeg -H 480 -W 640 -c 1

And now for the results of testing both the PS3Eye and Logitech C920

Here we see capturing 1000 frames from the Logitech C920 in MJPEG format takes ~33.6 seconds which is ~29.76 frames per second.

Here we see capturing 1000 frames from the Logitech C920 in YUYV format takes ~67 seconds which is ~14.92 frames per second.

Moving to the PS3Eye we see that if we try to capture at 30 FPS, we receive a select time out error, but if we set the frame rate to 15, we are successful.  If you compare the results of the PS3Eye capture with the results of the Logitech C920 YUYV test, you will see the real times are essentially the same, almost 15 frames per second.

At this point you maybe wondering why the Logitech C920 does not receive select timeouts at 30 FPS YUYV but the PS3Eye does.  If you notice, even though we set the frame rate to 30 FPS, we receive frames from the C920 at about 15 FPS.  The C920 use isochronous transfers as opposed to bulk like the PS3Eye, and isochronous guarantees transfer speed but not delivery.  It is likely that frames are getting dropped but enough make it through fast enough that we do not receive select timeouts.  I have not tested this further as of now.  For more information on USB transfers see [4].

In our final screenshot we can see that framegrabber uses very little cpu (.3%) while just grabbing frames.

I hope you find framegrabber useful.  The interested reader can extend the process_image function to do as they will with frames captured from the webcam.

[UPDATE 1]
It seems some MJPEG streams omit the Huffman table causing the resulting jpeg captures to fail to open in some programs^[5].  The error message, if any, is something along the lines of "Huffman table 0x00 was not defined".  If you cannot open the MJPEG captures, please try the Python script MJPEGpatcher.py below.  MJPEGpatcher should patch the captured jpeg with the required information.  It takes a single command line argument, -f FILENAME, and outputs FILENAME[patched].jpg.  The Logitech C920 does not exhibit this behavior.  MJPEGpatcher has been tested and works on images captured by a Toshiba webcam built into a laptop as well as an image submitted by a reader.  I would appreciate any feedback.

[UPDATE 2]
William C Bonner pointed out in the comments that I neglected to provide any timing information for the C920 for resolutions greater than 640x480.  When researching for this post, I was interested in explaining why the C920 could provide 640x480 at 30 FPS  and the PS3Eye could not.  In doing so, I focused on the greatest resolution and frame rate the two cameras had in common.  To redress my omission, here are timings for the C920 for 1920x1080 at 30 FPS in MJPEG, H264, and YUYV formats.  It can be seen below that the C920 is able to provide 1920x1080 at 30 FPS in both MJPEG and H264 formats, but YUYV tops out around 2.49 FPS

framegrabber.c (18.79 kb)
MJPEGpatcher.py (6.46 kb)

[1] https://github.com/twam/v4l2grab
[2] http://www.ideasonboard.org/uvc/
[3] http://linuxtv.org/downloads/v4l-dvb-apis/capture-example.html
[4] http://www.beyondlogic.org/usbnutshell/usb4.shtml
[5] http://www.the-labs.com/Video/odmlff2-avidef.pdf