Using toolkits

A toolkit is a collection of artifacts that are organized into a package. Toolkits make SPL or native functions and primitive or composite operators reusable across different applications. A toolkit provides one or more namespaces that contain functions, operators, and types that are packaged as part of the toolkit, all of which can then be reused in other Streams applications.

The Streams platform includes many toolkits, each with useful operators and functions. For example, the operators used to connect to Apache Kafka are in the Kafka toolkit. Almost all Streams applications that you create will use one or more external toolkits.

Toolkits and developing SPL applications

SPL code editing features (IntelliSense) are powered by the SPL language server, which relies on toolkits to provide content (e.g., operator details on hover). Toolkits are grouped into two categories: build service toolkits and local toolkits.

Build service toolkits

This set of toolkits depends on the Streams version of your default Streams instance (in the Streams Explorer).

Local toolkits

These toolkits are stored locally on your machine. The Toolkit Paths setting determines which toolkits are included.

Note: In order for a toolkit to be recognized, it must be already indexed (i.e., it must have an up-to-date toolkit.xml file).

Viewing and managing toolkits

To view and manage the toolkits that this extension is using:

  1. Bring up the Streams Explorer by clicking on the IBM Streams icon IBM Streams icon on the side of VS Code.
  2. Find the Toolkits section where the toolkit names and versions are listed. For information about managing toolkits, see Toolkits.

    Note: If the bundled toolkits are being used, the Build service section will display None available.

Toolkits in the Streams Explorer

Finding a toolkit

You can search for open-source Streams toolkits in GitHub or develop a custom toolkit.

Adding a toolkit

If your Streams application uses a toolkit that is not bundled with this extension or is not included as part of the Streams build service for the default Streams instance, you must add the toolkit locally or to the build service. Adding the toolkit to the build service is beneficial if the toolkit is large; you only need to upload the toolkit once rather than uploading the toolkit with every application build.

Note: In order for a toolkit to be recognized, it must be already indexed (i.e., it must have an up-to-date toolkit.xml file).

Adding a local toolkit

  1. Designate a folder on your machine where Streams toolkits will be stored. This folder will contain any toolkits that you want to use in your streaming applications.
  2. Place the toolkit to add in the designated toolkits folder.
  3. Update the Toolkit Paths setting to point to the designated toolkits folder (must be an absolute path). You can either:

Notes:

  • The Toolkit Paths setting accepts paths to multiple toolkits folders (comma or semicolon separated), but it is recommended to use a single folder.
  • Developing a custom toolkit in a folder that is part of the Toolkit Paths setting is not recommended. Only indexed toolkits should be specified in the setting.
Toolkit Paths setting

The Toolkits section in the Streams Explorer will display the toolkits in the toolkit paths under the Local node.

Local toolkits

Adding a toolkit to the Streams build service

Note: This is only available when the version of the default Streams instance is: IBM Cloud Pak for Data deployment or IBM Streams standalone deployment.

  1. Either:
    • Bring up the Command Palette and select Add Toolkit to Streams Build Service.
    • Bring up the Streams Explorer and expand the Toolkits section. Then, hover over the Build service node and click on the Add Toolkit to Streams Build Service icon Add Toolkit to Streams Build Service icon that appears on the right.
    • Bring up the Explorer and right-click on a toolkit folder that contains an indexed toolkit (with a toolkit.xml file) or a toolkit.xml file. Then, select Add Toolkit to Streams Build Service.
  2. In the dialog that appears, you can browse to a different toolkit folder or accept the selected toolkit folder. Click on the Set as toolkit folder button.

If the toolkit is successfully added, the Toolkits section in the Streams Explorer will display the new toolkit under the Build service node.

Build service toolkits

Removing a toolkit

If you no longer need a toolkit, you can remove it so that this extension no longer tracks it.

Removing a local toolkit

To remove a single toolkit, manually delete or move the toolkit from the toolkit folder. Bring up the Streams Explorer and expand the Toolkits section. Then, hover over the local toolkit node and click on the View Toolkit icon View Toolkit icon that appears on the right. This will reveal the toolkit in your OS file manager where you can delete or move the toolkit.

To remove an entire toolkit path, update the Toolkit Paths setting so that it no longer points to the toolkit path. You can either:

The Toolkits section in the Streams Explorer will no longer display the removed toolkit(s) under the Local node.

Removing a toolkit from the Streams build service

Note: This is only available when the version of the default Streams instance is: IBM Cloud Pak for Data deployment or IBM Streams standalone deployment.

  1. Either:
  2. Select one or more toolkits to remove.

If the toolkit(s) are successfully removed, the Toolkits section in the Streams Explorer will no longer display the toolkit(s) under the Build service node.

Using a toolkit

To use functions, operators, and types that are defined in a toolkit, configure your Streams application.

  1. Add the toolkit as a dependency by adding an entry in your project’s info.xml file. Example:
    <info:dependencies>
      <info:toolkit>
        <common:name>com.ibm.streams.sometoolkit</common:name>
        <common:version>[minVer,maxVer)</common:version>
      </info:toolkit>
    </info:dependencies>
    
  2. Import a function, operator, or type from the toolkit by adding a use directive at the top of your SPL file. Example:
    use com.ibm.streams.sometoolkit.somenamespace::SomeOperator;
    
  3. Use the imported function, operator, or type in your code. Example:
    stream<int32 myint> AnotherOperator = SomeOperator() {
      // Do something here
    }
    

When building your SPL application, toolkit dependencies are resolved automatically. If you are building from a Makefile, ensure that you specify the toolkit lookup paths.

Bundled toolkits

The following toolkits are bundled with this extension.

Name Description Version
com.ibm.streams.cep Provides the MatchRegex operator to perform complex event processing. 2.1.1
com.ibm.streams.cybersecurity Provides operators that are capable of analyzing DNS response records. 2.1.1
com.ibm.streams.geospatial Includes operators and functions that facilitate efficient processing and indexing of location data. 3.3.2
com.ibm.streams.pmml Provides operators to score input records using PMML models and to interact with the IBM Watson Machine Learning repository. 2.1.0
com.ibm.streams.rproject Includes the RScript operator, which you can use to run R commands and apply complex data mining algorithms to detect patterns of interest in data streams. 2.1.2
com.ibm.streams.rules Integrates IBM Operational Decision Manager (ODM) with IBM Streams to run business rules that are created in ODM. 2.1.2
com.ibm.streams.rulescompiler   1.2.19
com.ibm.streams.teda Provides a set of generic operators that are used in telecommunications applications, and it also provides an application framework that enables you to set up new file-to-file applications. 2.2.1
com.ibm.streams.text Includes operators to extracts information from text data. 2.3.2
com.ibm.streams.timeseries Includes operators and functions to condition, analyze, and model time series data. 5.0.1
com.ibm.streamsx.avro Supports serialization and deserialization of messages in an Apache Avro format. 1.2.2
com.ibm.streamsx.datetime Set of utilities to handle dates, times, and timestamps. 1.2.1
com.ibm.streamsx.dps Enables multiple applications running processing elements (PEs) on one or more machines to share application specific state information. 4.1.1
com.ibm.streamsx.elasticsearch Provides adapters for the Elasticsearch search and analytics engine. 2.1.1
com.ibm.streamsx.eventstore Provides adapters for the Db2 Event Store database. 2.0.3
com.ibm.streamsx.hbase Provides support for interacting with Apache HBase. 3.6.0
com.ibm.streamsx.hdfs Provides operators that can read and write data from Hadoop Distributed File System (HDFS) version 2 or later. 4.4.1
com.ibm.streamsx.inet Provides support for common internet protocol client functions and operators. 3.1.0
com.ibm.streamsx.iot Provides Internet of Things (IoT) integration with IBM Watson IoT Platform. 1.2.0
com.ibm.streamsx.jdbc Enables Streams applications to work with databases via JDBC. 1.6.0
com.ibm.streamsx.jms Provides operators and functions that help you interact with JMS systems such as IBM MQ or Apache ActiveMQ. 1.1.0
com.ibm.streamsx.json Provides standard transforms between SPL values and JSON objects. 1.5.2
com.ibm.streamsx.kafka Enables Streams applications to easily integrate with Apache Kafka. 1.9.4
com.ibm.streamsx.mail Provides support for the SMTP and IMAP protocols. 2.0.0
com.ibm.streamsx.messagehub Enables Streams applications to work with IBM Event Streams. 1.9.3
com.ibm.streamsx.messaging Includes operators and functions that help you interact with messaging systems such as Kafka, JMS, XMS, and MQTT. 5.4.1
com.ibm.streamsx.mqtt Enables Streams applications to work with MQTT providers. 1.0.1
com.ibm.streamsx.network Provides operators and functions for ingesting and parsing raw network data at the packet level. 3.2.1
com.ibm.streamsx.objectstorage Provides the ability to read and write data from and to IBM Cloud Object Storage, respectively. 1.9.2
com.ibm.streamsx.rabbitmq Provides operators that allows your Streams applications to read and send messages from RabbitMQ. 1.2.1
com.ibm.streamsx.sparkmllib Allows Apache Spark’s MLlib library to be used for real-time scoring of data. 1.2.0
com.ibm.streamsx.topology Provides mechanisms to build IBM Streams applications in other programming languages, with Java, Scala and Python support. 1.12.10
spl Provides a set of operators, types, and functions that are specific to IBM Streams. 1.4.0