############### streamsx-runner ############### ******** Overview ******** Submits or builds a Streams application to the Streaming Analytics service. The application to be submitted can be: * A Python application defined through :py:class:`~streamsx.topology.topology.Topology` using the ``--topology`` flag. * An SPL application (main composite) using the ``--main-composite`` flag. * A Streams application bundle (``sab`` file) using the ``--bundle`` flag. *************************** Streaming Analytics service *************************** The Streaming Analytics service is defined by: * Service name - ``--service-name`` defaulting to environment variable ``STREAMING_ANALYTICS_SERVICE_NAME``. The service name must exist in the vcap services. * Vcap services - Environment variable ``VCAP_SERVICES`` containing JSON representation of the service definitions or a file name containing the service definitions. ************** Job submission ************** Job submission occurs unless ``--create-bundle`` is set. *************** Bundle creation *************** When ``-create-bundle`` is specified with ``-main-composite`` or ``--topology`` then a Streams application bundle (sab file) is created. If environment variable `STREAMS_INSTALL` is set the the build is local otherwise the build occurs in the IBM Cloud using the Streaming Analytics service. When ``STREAMS_INSTALL`` is not set then `streamsx-runner` can be executed with no local Streams install. When compiling an SPL application (``--main-composite``) then the path to the application toolkit containing the main composite must be listed with ``--toolkits``. Any other required local toolkits must be listed with with ``--toolkits``. ***** Usage ***** .. code-block:: none streamsx-runner [-h] [--service-name SERVICE_NAME] | [--create-bundle] (--topology TOPOLOGY | --main-composite MAIN_COMPOSITE | --bundle BUNDLE) [--toolkits TOOLKITS [TOOLKITS ...]] [--job-name JOB_NAME] [--preload] [--trace {error,warn,info,debug,trace}] [--submission-parameters SUBMISSION_PARAMETERS [SUBMISSION_PARAMETERS ...]] [--job-config-overlays file] Execute a Streams application using a Streaming Analytics service. optional arguments: -h, --help show this help message and exit --service-name SERVICE_NAME Submit to Streaming Analytics service --create-bundle Create a bundle (sab file). No job submission occurs. --topology TOPOLOGY Topology to call --main-composite MAIN_COMPOSITE SPL main composite (namespace::composite_name) --bundle BUNDLE Streams application bundle (sab file) to submit to service Build options: Application build options --toolkits TOOLKITS [TOOLKITS ...] SPL toolkit path containing the main composite and any other required SPL toolkit paths. Job options: Job configuration options --job-name JOB_NAME Job name --preload Preload job onto all resources in the instance --trace {error,warn,info,debug,trace} Application trace level --submission-parameters SUBMISSION_PARAMETERS [SUBMISSION_PARAMETERS ...], -p SUBMISSION_PARAMETERS [SUBMISSION_PARAMETERS ...] Submission parameters as name=value pairs --job-config-overlays file Path to file containing job configuration overlays JSON. Overrides any job configuration set by the application. ***************************************** Submitting to Streaming Analytics service ***************************************** An application is submitted to a Streaming Analytics service using ``--service-name SERVICE_NAME``. The named service must exist in the VCAP services definition pointed to by the ``VCAP_SERVICES`` environment variable. The application is submitted as source (except ``--bundle``) and compiled into a Streams application bundle (``sab`` file) using the build service before being submitted as a running job to the service instance. .. seealso:: :ref:`sas-access` .. _runner-python-apps: Python applications =================== To submit a Python application a Python function must be defined that returns the application (and optionally its configuration) to be submitted. The fully qualified name of this function is specified using the ``--topology`` flag. For example, an application can be submitted as:: streamsx-runner --service-name Streaming-Analytics-xd \ --topology com.example.apps.sensor_ingester The function returns one of: * a :py:class:`~streamsx.topology.topology.Topology` instance defining the application * a ``tuple`` containing two values, in order: * a :py:class:`~streamsx.topology.topology.Topology` instance defining the application * job configuration, one of: * :py:class:`~streamsx.topology.context.JobConfig` instance * ``dict`` corresponding to the configuration object passed into :py:func:`~streamsx.topology.context.submit` For example the above function might be defined as:: def _create_sensor_ingester_app(): topo = Topology('SensorIngesterApp') # Application declaration omitted ... return topo def sensor_ingester(): return (_create_sensor_ingester_app(), JobConfig(job_name='SensorIngester')) Thus when this application is submitted using the `sensor_ingester` function it is always submitted with the same job name `SensorIngester`. The function must be accessible from the current Python path (typically through environment variable ``PYTHONPATH``). .. _runner-spl-apps: SPL applications ================ The main composite that defines the application is specified using the ``-main-composite`` flag specifing the fully namespace qualified name. Any required local SPL toolkits, *including the one containing the main composite*, must be indivdually specified by location to the ``--toolkits`` flag. Any SPL toolkit that is present on the IBM Cloud service need not be included. For example, an application that uses the Slack toolkit might be submitted as:: streamsx-runner --service-name Streaming-Analytics-xd \ --main-composite com.example.alert::SlackAlerter \ --toolkits $HOME/app/alerters $HOME/toolkits/com.ibm.streamsx.slack where ``$HOME/app/alerters`` is the location of the SPL application toolkit containing the ``com.example.alert::SlackAlerter`` main composite. .. warning:: The main composite name must be namespace qualified. Use of the default namespace for a main composite is not recommended as it increases the chance of a name clash with another SPL toolkit. .. _runner-sab: Streams application bundles =========================== A Streams application bundle is submitted to a service instance using ``--bundle``. The argument to ``--bundle`` is a locally accessible file that will be uploaded to the service. The bundle must have been created on using an IBM Streams install whose architecture and OS version matches the service instance. Currently this is ``x86_64`` and RedHat/CentOS 6 or 7 depending on the service instance. The ``--toolkits`` flag must not be specified when submitting a bundle. Job options =========== Job options, such as ``--job-name``, configure the running job. For ``--topology`` job options set as arguments to ``streamsx-runner`` override any configuration returned from the function defining the application. ************************************ Creating Streams application bundles ************************************ ``--create-bundle`` uses a local IBM Streams install to attempt to mimic the build that would occur with ``-topology`` or ``--main-composite``. Differences between the local environment and the IBM Cloud Streaming Analytics build environment may cause build failures in one and not the other. This can be used as a mechanism to perform a local test build before using the service, or as a valid mechanism to create bundles for later upload with ``--bundle``. For example simply changing the ``--service-name name`` to ``--create-bundle`` perfoms a local build of the same application:: # Submit to an Streaming Analytics service streamsx-runner --service-name Streaming-Analytics-xd \ --main-composite com.example.alert::SlackAlerter \ --toolkits $HOME/app/alerters $HOME/toolkits/com.ibm.streamsx.slack # Build the same application locally streamsx-runner --create-bundle \ --main-composite com.example.alert::SlackAlerter \ --toolkits $HOME/app/alerters $HOME/toolkits/com.ibm.streamsx.slack