Dependent Python packages

Toolkits > com.ibm.streamsx.topology 2.1.0 > com.ibm.streamsx.topology.python > Creating SPL Operators from Python code > Dependent Python packages

The decorated Python classes and functions used for SPL operators can depend on additional Python packages or modules.

Dependent Python packages can either be stored in the toolkit or installed using pip during SPL compilation.

Packages & modules stored in the SPL toolkit

These directories in a toolkit are automatically added to the Python search path during execution of an operator.
  • opt/python/streams - Contains modules that define Python callables that will be created as SPL operators
  • opt/python/packages - Root directory for Python packages hierarchy.
  • opt/python/modules - Arbitrary modules, not defined as a packages.

By default all of these directories are included in the Streams application bundle (sab) for a compiled application.

Thus any packages or modules stored in those directories are part of the toolkit, thus increasing the distribution size of the toolkit.

Packages installed by pip at SPL compile time

Python packages can be explictly requested to be installed during the SPL compilation using pip3 install --user. The packages are installed locally into the compliation's output directory, thus becoming part of the Streams application bundle (sab). The packages will be installed from PyPi.

During SPL compilation (sc) pip must be in the user's PATH.

Any required packages and their dependencies are added to the sab if they are not globally installed. Thus there is an assumption that the compile machines and runtime machines have an identical set of globally installed packages. This is guaranteed for the Streaming Analytics service on IBM Cloud.

For example is there was a dependency on numpy and geocoder and numpy was globally installed then only geocoder would be installed into the compliation's output directory (in addition to any dependencies geocoder has that were not globally installed).

Packages are explictly requested for the whole toolkit or a specific module.

Any packages the modules (under opt/python/streams) depend on when being loaded, must be accessible at operator extraction time. For example, if the module opt/python/streams/geo_ops.py directly imports geocoder for use in its decorated functions or classes then since geo_ops.py is loaded to determine the SPL primitive operators, geocoder must be accessible to the Python environment running the extraction script.

Tookit wide requirements

If the file opt/python/streams/requirements.txt exists in the toolkit then it is taken as a pip requirements file. During SPL compilation it will be passed to pip using the -requirement flag for installation of the required packages into the output directory.

See: https://pip.pypa.io/en/stable/user_guide/#requirements-files

When a toolkit with a requirements.txt is built using the build service then its required packages are installed remotely and made available to the extraction script.

Module specific requirements

A module containing SPL primitive operator decorated clases and functions can specify a list of packages (or more precisely requirement specifiers) using the function spl_pip_packages. 

# Packages geocoder and python-goehash will be installed
# into the sab for the application invoking any operator
# in this module unless they are globally installed at
# SPL (sc) compile time.
def spl_pip_packages():
    return ['geocoder', 'python-geohash']