Creating SPL Operators from Python code

Toolkits > com.ibm.streamsx.topology 1.5.13.__dev__ > com.ibm.streamsx.topology.python > Creating SPL Operators from Python code

SPL operators that call a Python function or callable class are created by decorators provided by this toolkit.

bin/spl-python-extract.py is a Python script that creates SPL operators from Python functions and classes contained in modules in a toolkit. The resulting operators embed the Python runtime to allow stream processing using Python.

To create SPL operators from Python functions or classes one or more Python modules are created in the opt/python/streams directory of an SPL toolkit.

Each module must import the streamsx.spl package which is located in the opt/packages directory of this toolkit. It contains the decorators use to create SPL operators from Python functions. The module must also define a function splNamespace that returns a string containing the SPL namespace the operators for that module will be placed in.

For example:

# Import the SPL decorators
from streamsx.spl import spl

# Defines the SPL namespace for any functions in this module
# Multiple modules can map to the same namespace
def splNamespace():
   return "com.ibm.streamsx.topology.pysamples.mail"

Decorating a Python class produces a stateful SPL operator. The instance fields of the class are the state for the operator. Any parameters to the __init__ method (exluding the first self parameter) are mapped to operator parameters.

Decorating a Python function produces a stateless SPL operator. The function may reference variables in the module that are effectively state but such variables are shared by all invocations of the operator within the same processing element.

Any Python docstring for the function or class is copied into the SPL operator's description field in its operator model, providing a description for IDE developers using the toolkit.

Functions or classes in the modules that are not decorated, decorated with @spl.ignore or start with spl are ignored and will not result in any SPL operator.

Python classes as SPL operators

Decorating a Python class creates a stateful SPL operator where the instance fields of the class are the operator's state.

Python functions as SPL operators

Decorating a Python function creates a stateless SPL operator.

Processing SPL tuples in Python

SPL tuples are converted to Python objects and passed to a decorated callable.

Submission of SPL tuples from Python

The return from a decorated callable results in submission of SPL tuples on the associated outut port.

Supported SPL types

A limited set of SPL types are supported.

@spl decorator options

@spl decorators support an number of options.

@spl.source

Decorator to create a stateful or stateless SPL source operator from a Python iterable class or function.

@spl.filter

Decorator to create a stateful or stateless SPL filter operator from a Python callable class or function.

@spl.map

Decorator to create a stateful or stateless SPL map operator from a Python callable class or function.

@spl.for_each

Decorator to create a stateful or stateless SPL sink operator from a Python callable class or function.

@spl.pipe

Decorator to create a stateless SPL map operator from a Python function.

@spl.sink

Decorator to create a stateless SPL sink operator.

@spl.ignore

Decorator to ignore a Python function.

Extracting SPL operators from Python

Overview: To create SPL operators in a toolkit, execute: python3 spl-python-extract.py -i toolkit-directory Any Python module in the toolkit's opt/python/streams directory will have its decorated classes and functions converted to SPL operators.

Sample toolkit

A toolkit with a number of decorated Python functions and SPL applications that invoke them is supplied under samples/python/com.ibm.streamsx.topology.pysamples. Executing applications using Python: PYTHONHOME: The location of Python at runtime is defined by the environment variable PYTHONHOME.