Operators and Types

All of the operators and types in this toolkit are defined in the Streams namespace ’com.ibm.streamsx.opencv’. To use them in an SPL application, include this statement at the top of your SPL or SPLMM source file:

    use com.ibm.streamsx.opencv::*;

For sample applications that use these operators and types, see Sample Applications.

For suggestions on using the tookit, see Using the OpenCV Toolkit.

For instructions on installing the toolkit and its dependencies, see Installing the OpenCV Toolkit.

The Streams types and operators in this toolkit are ’wrappers’ for C/C++ data structures, functions, and classes defined in the FFmpeg and OpenCV libraries. For further information on the OpenCV API, please refer to the PDF files installed in /usr/local/share/OpenCV/doc and the online OpenCV documentation. For further information on the FFmpeg API, please refer to the ’ffmpeg’ man pages and the online FFmpeg documentation.

IplImage and BoxList types

All of the operators in this toolkit use the Streams type IplImage to represent video frames in tuples. It is defined like this:

    type IplImage = 
        int32 ipl_channels, 
        int32 ipl_depth, 
        int32 ipl_width, 
        int32 ipl_height, 
        list<uint8> ipl_data, 
        uint64 tag ;

The Streams type IplImage is a ’wrapper’ for the . Individual images representing video frames are held in the ’ipl_data’ attribute of IplImage. The format of image data is described in the .

Some of the operators in this toolkit use the Streams type BoxList to represent rectangular areas within a video frame that are of particular interest. It is defined like this:

    type BoxList = 
        list<int32> x_list, 
        list<int32> y_list, 
        list<int32> width_list, 
        list<int32> height_list ;

An attribute of Streams type BoxList always appears in tuples along with an attribute of Streams type IplImage. The BoxList contains four integer lists that always have the same number of elements, representing the position and size of a rectangular subset of the video frame in the ’ipl_data’ attribute of IplImage.

The Crop and CropBoxes operators can produce tuples of type IplImage or CroppedIplImage:

    type CroppedIplImage = 
        int32 ipl_channels, 
        int32 ipl_depth, 
        int32 ipl_width, 
        int32 ipl_height, 
        int32 ipl_x, 
        int32 ipl_y, 
        list<uint8> ipl_data, 
        uint64 tag ;

The attributes ’ipl_x’ and ’ipl_y’ of type CroppedIplImage are assigned the original position of the cropped image. That is, ’ipl_x’ and ’ipl_y’ are the ’x’ and ’y’ values the output image had in the input image.

AddText operator

Draws the value of the parameter ’text’ on the image, in the specified color, at the specified coordinates. Newline characters in the text are handled properly, but other non-printing characters are rendered as question marks.

Input:

Output:

Parameters:

Used in these sample applications:

Apply2 operator

Apply a specific OpenCV function on a set of images. The operator waits until it has received an image on each of its input ports then it applies the function ’f’ as follow:

f(...f(f(img[0],img[1]),img[2])...,img[n-1])

If too many images are recieved for a given port (for example if there is a large rate mismatch) the operator will terminate.

Input:

Output:

Parameters:

Used in these sample applications:

Average operator

Compute the average luminosity of all the pixels in the image and print the result on the image. If the input image is RGB, it will first be converted to black&white before computing the average. The output tuple contains a copy of the original image (RGB or B&W) with the average value.

Input:

Output:

Parameters: none

Used in these sample applications:

BitBucket operator

Track number of tuples and their combined size from a particular stream. Upon receiving a punctuation on the stream, the stats {tuples/s, bytes/s, and number of seconds} are written to a log. A reset of time, #tuples, and #bytes occurs with each punctuation also.

Input:

Output: none

Parameters: none

Used in these sample applications:

Blank operator

Generate blank frames of a given size and depth, with a specific color. The rate at which the frames are sent is specified. It is possible to limit how many frames are sent total.

Input: none

Output:

Parameters:

Used in these sample applications:

Canny operator

Apply the ’cvCanny" function to the input image and send the result out. This operation can be used to detect contours of objects in the image.

Input:

Output:

Parameters:

Used in these sample applications:

CaptureFromCAM operator

Capture video frames using a camera attached to the host. OpenCV supports V4Linux and FireWire on Linux. The list of supported cameras depends on the version of Linux and the type of device. Cameras that are compatible with the UVC standard are good candidates since they do not require a specific driver to be supported.

Input: none

Output:

Parameters:

Used in these sample applications:

CaptureFromFile operator

Read video frames from a file. The list of supported formats varies depending on the options used during the build of OpenCV and FFmpeg.

Input: none

Output:

Parameters:

Used in these sample applications:

CaptureFromURL operator

Obtain successive video frames from a HTTP server. The URL provided will be queried periodically to obtain the next frame. This is ideal to get video frames from an IP camera or a traffic camera. libCURL is used to retrieve the images. Please refer to libCURL documentation for the supported URL formats.

The format of the image fetched can be any format supported by OpenCV. This operator is not able to process web pages. The URL must be an image. The same URL is used to query successive frames and it is assumed that the image changes periodically on the server.

Input: none

Output:

Parameters:

Used in these sample applications:

CodeBook operator

Compute the foreground/background segmentation of each frames using the CodeBook algorithm described in the book **Learning OpenCV”. This operator learns about the background for a period of time and builds a statistical model of each pixel. Once the learning phase ends, it uses the model to label each pixel of subsequence frames. A pixel belongs either to the forground or background. The result is a binary image where each pixel is either ’0’ (background) or ’255’ (foreground). Once the learning phase ends, the statistical models are not updated.

This algorithm is more powerful than a typical frame difference since the model built can accomodate periodic variation like water falls, tree leaves movement, etc. However during the learning phase, there must not be any foreground object in the field.

Input:

Output:

Parameters:

Used in these sample applications:

Collage operator

Assemble several video streams in a side by side fashion and produce a new video stream. Each video stream must be received on a different port. The height, depth and number of channels of the frame of each video stream must be identical. The frames are assembled from left to right. Each incoming video stream must be producing frames at the same rate. The operator will wait for a frame on each port before emitting a new composite frame.

           +----------+----------+----------+
           |          |          |          |
           |  img #1  |  img #2  |  img #3  |
           |          |          |          |
           +----------+----------+----------+

Input:

Output:

Parameters: none

Used in these sample applications:

CombineImg operator

Overlay an image over another. The result image size is always the same as the size of the image on the first input port. The size of the second image may be bigger or smaller than the first image. The result is computed by computing the merge value of all the pixels overlapping between both images. The pixels from the second image that are outside the overlaping zone are ignored. The pixels of the first image that are not in the overlaping zone are unchanged. The opacity can be adjusted between 0 and 100 of the second image.

             +----------------------------------------+
             |                                        |
             |                  img #1                |
             |                                        |
             |                                        |
             |           + - - - - - - - - - - - - - -+- - - - +
             |                                        |
             |           |                            |        |
             |                                        |
             |<-offset ->|      [overlap zone]        |        |
             |                                        |
             |           |                            |        |
             |                                        |
             +-----------+----------------------------+        |

                         |                img #2               |

                         + - - - - - - - - - - - - - - - - - - +

Input:

Output:

Parameters:

Used in these sample applications:

ConnectedComponents operator

Compute the contours of each frames then approximate contours into individual components discarding small components. This uses an algorithm described in the book **Learning OpenCV”.

Input:

Output:

Parameters:

Used in these sample applications:

Crop operator

Crops the input image as specified by the ’topleft’ and ’bottomright’ parameters, and outputs the cropped image. Cropping rectangle must be inside of the input image.

Input:

Output:

Parameters:

Used in these sample applications:

CropBoxes operator

Crops the input image as specified by the input BoxList (a list of zero or more rectangles), and outputs zero or more cropped images. Cropping rectangles must be inside of the input image.

Input:

Output:

Parameters: none

Used in these sample applications:

CvtColor operator

Convert the color of the input image. Constants for color conversion are in cv.h header file. Common constants include: 'CV_GRAY2BGR', 'CV_BGR2GRAY', and 'CV_RGB2GRAY'.

Input:

Output:

Parameters:

Used in these sample applications:

Dilate operator

Dilates the source image using cvDilate function.

Input:

Output:

Parameters:

Used in these sample applications:

DisplayProxy operator

Output the image stream over a socket connection on a specific port. There are two operation modes, one that sends to a specific hostname or one that listens for an incoming connection and then sends on that connection. Hence either a hostname parameter must be set or the listen parameter set to true.

Input:

Output: none

Parameters:

Used in these sample applications:

DoLog operator

Logarithmic transform of the input image (pixel-by-pixel). Common use is to enhance low pixel values at the expense of loss of information in the high pixel values.

Input:

Output:

Parameters:

Used in these sample applications:

DrawCircles operator

Draws circles at a the set of locations specified in the BoxList. Centers of the circles are determined by taking the (x,y) coordinate and adding 1/2 the width or height respectively. Radius is 1/4 the sum of width and height.

Input:

Output:

Parameters:

Used in these sample applications:

DrawContours operator

Draws the input image contours in white on top of a black output image.

Input:

Output:

Parameters: none

Used in these sample applications:

DrawRectangles operator

Draws rectangles at a the set of locations specified in the BoxList. Top-left corner is determined by taking the (x,y) coordinate. The bottom-right corner is determined by adding the width or height respectively.

Input:

Output:

Parameters:

Used in these sample applications:

EqualizeHist operator

Equalizes the histogram of each channel in the source image using cvEqualizeHist function.

Input:

Output:

Parameters: none

Used in these sample applications:

Erode operator

Erodes the source image using cvErode function.

Input:

Output:

Parameters:

Used in these sample applications:

FaceDetection operator

Uses a face model to detect faces in images, then outputs the image with a list of boxes (one for each face found).

Input:

Output:

Parameters:

Used in these sample applications:

Flip operator

Flips images horizontal, vertical, or both.

Input:

Output:

Parameters:

Used in these sample applications:

FramePacer operator

Paces the images on the input port when sending out the output port. This is either based on a rate from an input video file or based on an integer parameter.

Input:

Output:

Parameters:

Used in these sample applications:

HTTPViewer operator

HTTP Server serving images at a static location. As new image tuples arrive the served image is changed.

If viewed using a browser, clicking refresh should update the displayed image.

Input:

Output: none

Parameters:

Used in these sample applications:

MakeBox operator

Draws rectangles at a the set of locations specified in the BoxList. Top-left corner is determined by taking the (x,y) coordinate. The bottom-right corner is determined by adding the width or height respectively.

Input:

Output:

Parameters:

Used in these sample applications:

MJPEGStreamer operator

HTTP Server streaming images at a static location. As new image tuples arrive the image is sent to connected clients.

If viewed using a browser, the image should automatically update as new images arrive.

Input:

Output: none

Parameters:

Used in these sample applications:

MotionDetection operator

Detects object motion in the image and draws flow arrows on image. This runs the Shi and Tomasi algorithm to detect feature points in the input image. Then using the Pyrimidal Lucas Kanade Optical Flow algorithm determine the flow of points between the previous image and the input image, resulting in a new set of feature points between the two images. Using the results flow arrows are drawn on the input image, longer arrows indicates larger movement.

Input:

Output:

Parameters:

Used in these sample applications:

Nop operator

No operation is performed on the input image before outputing it.

Input:

Output:

Parameters: none

Used in these sample applications:

RankFilter operator

Filters input image pixel-by-pixel by comparing it to its neighboring pixels within a radius. The comparison/replacement method is one of the following: minimum, median, or maximum.

The median filter is normally used to reduce noise in an image while preserving image details.

Input:

Output:

Parameters:

Used in these sample applications:

ReadImage operator

Reads the image at the path indicated in the input tuple, then outputs the image as a tuple.

Input:

Output:

Parameters: none

Used in these sample applications:

Resize operator

Resizes the input image and outputs the result.

Input:

Output:

Parameters:

Used in these sample applications:

Rotate operator

Scales the input image and rotates the image around a center point.

Input:

Output:

Parameters:

Used in these sample applications:

SaveImage operator

Save each input image in an image file.

The file format is deduced from the file extension. The image formats supported depend upon the image codecs installed on the system, which probably include:

For details, see the documentation for the

Optionally, a sequential integer can be inserted in front of the file extension.

Input:

Output (optional):

Parameters:

Used in these sample applications:

SaveToFile operator

Save the input images to a video file. The video file frame rate can be defined either by parameter ’rate’ or by the frame rate of another video file ’frate’.

The codec for the output video file is defined by the ’fourcc’ parameter. Common examples are:

   ##   Codec ----->  CV_FOURCC
   ##
   ##   mpeg1         PIM1
   ##   motion-jpeg   MJPG
   ##   mpeg4.2       MP42
   ##   mpeg4.3       DIV3
   ##   mpeg4         DIVX
   ##   h263          I263
   ##   flv1          FLV1

Input:

Output: none

Parameters:

Used in these sample applications:

Sequencer operator

Uses an integer tag on incoming images to order the images in ascending order.

Input:

Output:

Parameters: none

Used in these sample applications:

SkipEmpty operator

Discards empty frames in a stream. A frame is determined to be non-empty if the non-black pixel content is above a threshold.

Input:

Output:

Parameters:

Used in these sample applications:

SkipFrame operator

Discards frames in a stream. A frame is skipped either adaptively and/or fixed. If ’fixed’, a ’frame’ number of images are skipped between each processed frame. If ’adaptive’, a fixed number of images can still be skipped if the ’frame’ parameter is specified, regardless adaptive will also skip any frames when the previous frame has not finished being output yet.

Input:

Output:

Parameters:

Used in these sample applications:

Smooth operator

Use guaussian smoothing to smooth images.

Input:

Output:

Parameters:

Used in these sample applications:

Snow operator

Generate random **snow” frames consisting of either black or white pixels. The rate at which the frames are sent is specified.

Input: none

Output:

Parameters:

Used in these sample applications:

Sobel operator

Apply the Sobel edge detection function to the input image, and output the resulting image.

Input:

Output:

Parameters: none

Used in these sample applications:

SourceProxy operator

Input the image stream over a socket connection on a specific port. There are two operation modes, one that receives from a specific hostname or one that listens for an incoming connection and then receives on that connection. Hence either a hostname parameter must be set or the listen parameter set to true.

Input: none

Output:

Parameters:

Used in these sample applications:

Stats operator

Prints incoming stream stats on images before outputing. Parameters allow for specifying which stats are printed, by default all available stats are printed (FPS, KBs, image size, number of channels, and count.)

Input:

Output:

Parameters:

Used in these sample applications:

Tagger operator

Tags incoming images with an ascending integer tag.

Input:

Output:

Parameters: none

Used in these sample applications:

Threshold operator

Applies a fixed-level threshold to input image. Images must be single-channel, 8-bit images. Threshold types can be found in cv.h, common types are: 'CV_THRESH_TOZERO', 'CV_THRESH_BINARY' and 'CV_THRESH_TOZERO_INV'

Input:

Output:

Parameters:

Used in these sample applications:

TimeStamp operator

Prints a timestamp on input images before outputting.

Input:

Output:

Parameters:

Used in these sample applications:

TrackBar operator

Creates a Trackbar to allow users to input a sliding integer value at runtime. The Trackbar can attach to an existing window or be in a window of its own.

This operator is currently setup to interact with 'Canny', 'MotionDetection', and 'Threshold' operators. Each of these operators require integer inputs either via a Trackbar or a parameter. To use a Trackbar these operators expect the tuple to contain an integer parameter whose name is 'trackbar_{parameter name}' (replacing 'parameter name' with the appropriate name, e.g. 'trackbar_threshold1' or 'trackbar_featurecount')

Input: none

Output:

Parameters:

Used in these sample applications:

X11Viewer operator

Displays incoming images in an X11 window.

Input:

Output: none

Parameters:

Used in these sample applications:

© Copyright 2012, 2016, International Business Machines Corporation, All Rights Reserved