Develop IBM Streams applications with Python.
Overview
A functional api to develop streaming applications for IBM Streams using Python. Streams are defined, transformed and sinked (terminated) using Python functions. The return of a function determines the content of the stream. Tuples on a stream are Python objects, a stream may contain different types of objects.
Prerequites
- Install and configure IBM Streams Version 4.0.1 (or later).
- Install Python
- Either install Anaconda 3.5 - this is the easiest option. When running distributed the Streams instance application environment variable PYTHONHOME must be set to the install location.
- or install CPython 3.5.0 (or later). This is more involved and usually requires building Python from source code.
- Download and extract the IBM Streams Topology Toolkit, which includes the Python Application API.
- Include the fully qualified path of the com.ibm.streamsx.topology/opt/python/packages directory in the PYTHONPATH environment variable.
HelloWorld Application
Example code that builds and then submits a Hello World topology.
import mymodule;
from streamsx.topology.topology import *
import streamsx.topology.context
topo = Topology("HelloWorld")
hw = topo.source(mymodule.hello_world)
hw.sink(print)
streamsx.topology.context.submit("STANDALONE", topo.graph)
The source function is passed a callable that returns an Iterable, in this case mymodule.hello_world.
def hello_world() :
return ["Hello", "World!"]
The callable will be called when the application starts, and the SPL runtime will create an iterator from the returned value. Then each value returned from the iterator will be sent on the stream.
The sink function is passed a callable that will be called for each tuple on the stream, in this case the builtin print function.
User-supplied functions to operations
Operations such as
source and
sink accept a callable as input. The callable must be one of the following:
- the name of a built-in function
- the name of a function defined at the top level of a module
- an instance of a callable class defined at the top level of a module that implements the function __call__ and is picklable. Using a callable class allows state information such as user-defined parameters to be stored during class initialization and utilized when the instance is called.
The modules containing the callables, along with third-party libraries required by the modules, are copied into the Streams Application Bundle (sab file).
- Dependent libraries can be individual modules or packages.
- Dependent libraries can be installed in site packages, or not installed and simply reside in a directory in the Python search path
- Dependent native libaries outside of the package directory are not copied into the bundle
Limitations on callable inputs to operations:
- Callables must be defined in a module file.
- Callables must not be dynamically created nor anonymous.
- Callables must not be defined in the __main__ module. They must be defined in a separate module.
- Importing modules that contain user-defined functions with importlib is unsupported. The PYTHONPATH or sys.path must contain the directory where modules to import are located.
- Importing modules that contain user-defined functions from zip/egg/wheel files is unsupported.
To avoid name conflicts, do not create modules with the same name as
- streamsx which is used by the Python Application API
- built-in or standard library module names
- Python Application API functions
- Functions to create topologies and streams.
- Sample Application
- Example code that builds and then submits a simple topology.
- Context Types
- Only submission using DISTRIBUTED, BUNDLE and STANDALONE context type are supported.
- MQTT support
- Publishing and subscribing to an MQTT broker.