Using toolkits

A toolkit is a collection of artifacts that are organized into a package. Toolkits make SPL or native functions and primitive or composite operators reusable across different applications. A toolkit provides one or more namespaces that contain functions, operators, and types that are packaged as part of the toolkit, all of which can then be reused in other Streams applications.

The Streams platform includes many toolkits, each with useful operators and functions. For example, the operators used to connect to Apache Kafka are in the Kafka toolkit. Almost all Streams applications that you create will use one or more external toolkits.

Toolkits and developing SPL applications

SPL code editing features (IntelliSense) are powered by the SPL language server, which relies on toolkits to provide content (e.g., operator details on hover). Toolkits are grouped into two categories: build service toolkits and local toolkits.

Build service toolkits

This set of toolkits depends on the Streams version of your default Streams instance (in the Streams Explorer).

• If your default instance is an IBM Cloud Pak for Data deployment or IBM Streams standalone deployment, then this extension uses the toolkits that are available for build processing from the instance’s remote build service.
• If your default instance is an IBM Streaming Analytics service on IBM Cloud or there are no instances, then this extension uses the bundled toolkits.

Local toolkits

These toolkits are stored locally on your machine. The Toolkit Paths setting determines which toolkits are included.

Note: In order for a toolkit to be recognized, it must be already indexed (i.e., it must have an up-to-date toolkit.xml file).

Viewing and managing toolkits

To view and manage the toolkits that this extension is using:

1. Bring up the Streams Explorer by clicking on the IBM Streams icon on the side of VS Code.

2. Find the Toolkits section where the toolkit names and versions are listed. For information about managing toolkits, see Toolkits.

   Note: If the bundled toolkits are being used, the Build service section will display None available.

Finding a toolkit

You can search for open-source Streams toolkits in GitHub or develop a custom toolkit.

Adding a toolkit

If your Streams application uses a toolkit that is not bundled with this extension or is not included as part of the Streams build service for the default Streams instance, you must add the toolkit locally or to the build service. Adding the toolkit to the build service is beneficial if the toolkit is large; you only need to upload the toolkit once rather than uploading the toolkit with every application build.

Note: In order for a toolkit to be recognized, it must be already indexed (i.e., it must have an up-to-date toolkit.xml file).

Adding a local toolkit

1. Designate a folder on your machine where Streams toolkits will be stored. This folder will contain any toolkits that you want to use in your streaming applications.
2. Place the toolkit to add in the designated toolkits folder.
3. Update the Toolkit Paths setting to point to the designated toolkits folder (must be an absolute path). You can either:
   • Use the Settings editor.
   • Bring up the Command Palette and select Set IBM Streams Toolkit Paths.
   • Bring up the Streams Explorer and expand the Toolkits section. Then, hover over the Local node and click on the Add Toolkit Path icon that appears on the right.

Notes:

• The Toolkit Paths setting accepts paths to multiple toolkits folders (comma or semicolon separated), but it is recommended to use a single folder.
• Developing a custom toolkit in a folder that is part of the Toolkit Paths setting is not recommended. Only indexed toolkits should be specified in the setting.

The Toolkits section in the Streams Explorer will display the toolkits in the toolkit paths under the Local node.

Adding a toolkit to the Streams build service

Note: This is only available when the version of the default Streams instance is: IBM Cloud Pak for Data deployment or IBM Streams standalone deployment.

1. Either:
   • Bring up the Command Palette and select Add Toolkit to Streams Build Service.
   • Bring up the Streams Explorer and expand the Toolkits section. Then, hover over the Build service node and click on the Add Toolkit to Streams Build Service icon that appears on the right.
   • Bring up the Explorer and right-click on a toolkit folder that contains an indexed toolkit (with a toolkit.xml file) or a toolkit.xml file. Then, select Add Toolkit to Streams Build Service.
2. In the dialog that appears, you can browse to a different toolkit folder or accept the selected toolkit folder. Click on the Set as toolkit folder button.

If the toolkit is successfully added, the Toolkits section in the Streams Explorer will display the new toolkit under the Build service node.

Removing a toolkit

If you no longer need a toolkit, you can remove it so that this extension no longer tracks it.

Removing a local toolkit

To remove a single toolkit, manually delete or move the toolkit from the toolkit folder. Bring up the Streams Explorer and expand the Toolkits section. Then, hover over the local toolkit node and click on the View Toolkit icon that appears on the right. This will reveal the toolkit in your OS file manager where you can delete or move the toolkit.

To remove an entire toolkit path, update the Toolkit Paths setting so that it no longer points to the toolkit path. You can either:

• Use the Settings editor.
• Bring up the Command Palette and select Set IBM Streams Toolkit Paths.
• Bring up the Streams Explorer and expand the Toolkits section. Then, hover over the Local node and click on the Remove Toolkit Path icon that appears on the right. Select one or more toolkit paths to remove.

The Toolkits section in the Streams Explorer will no longer display the removed toolkit(s) under the Local node.

Removing a toolkit from the Streams build service

Note: This is only available when the version of the default Streams instance is: IBM Cloud Pak for Data deployment or IBM Streams standalone deployment.

1. Either:
   • Bring up the Command Palette and select Remove Toolkits from Streams Build Service.
   • Bring up the Streams Explorer and expand the Toolkits section. Then, hover over the Build service node and click on the Remove Toolkits from Streams Build Service icon that appears on the right.
2. Select one or more toolkits to remove.

If the toolkit(s) are successfully removed, the Toolkits section in the Streams Explorer will no longer display the toolkit(s) under the Build service node.

Using a toolkit

To use functions, operators, and types that are defined in a toolkit, configure your Streams application.

1. Add the toolkit as a dependency by adding an entry in your project’s info.xml file. Example:

   <info:dependencies>
     <info:toolkit>
       <common:name>com.ibm.streams.sometoolkit</common:name>
       <common:version>[minVer,maxVer)</common:version>
     </info:toolkit>
   </info:dependencies>
   

2. Import a function, operator, or type from the toolkit by adding a use directive at the top of your SPL file. Example:

   use com.ibm.streams.sometoolkit.somenamespace::SomeOperator;
   

3. Use the imported function, operator, or type in your code. Example:

   stream<int32 myint> AnotherOperator = SomeOperator() {
     // Do something here
   }
   

When building your SPL application, toolkit dependencies are resolved automatically. If you are building from a Makefile, ensure that you specify the toolkit lookup paths.

Bundled toolkits

The following toolkits are bundled with this extension.

Name	Description	Version
com.ibm.streams.cep	Provides the MatchRegex operator to perform complex event processing.	2.1.1
com.ibm.streams.cybersecurity	Provides operators that are capable of analyzing DNS response records.	2.1.1
com.ibm.streams.geospatial	Includes operators and functions that facilitate efficient processing and indexing of location data.	3.3.2
com.ibm.streams.pmml	Provides operators to score input records using PMML models and to interact with the IBM Watson Machine Learning repository.	2.1.0
com.ibm.streams.rproject	Includes the RScript operator, which you can use to run R commands and apply complex data mining algorithms to detect patterns of interest in data streams.	2.1.2
com.ibm.streams.rules	Integrates IBM Operational Decision Manager (ODM) with IBM Streams to run business rules that are created in ODM.	2.1.2
com.ibm.streams.rulescompiler		1.2.19
com.ibm.streams.teda	Provides a set of generic operators that are used in telecommunications applications, and it also provides an application framework that enables you to set up new file-to-file applications.	2.2.1
com.ibm.streams.text	Includes operators to extracts information from text data.	2.3.2
com.ibm.streams.timeseries	Includes operators and functions to condition, analyze, and model time series data.	5.0.1
com.ibm.streamsx.avro	Supports serialization and deserialization of messages in an Apache Avro format.	1.2.2
com.ibm.streamsx.datetime	Set of utilities to handle dates, times, and timestamps.	1.2.1
com.ibm.streamsx.dps	Enables multiple applications running processing elements (PEs) on one or more machines to share application specific state information.	4.1.1
com.ibm.streamsx.elasticsearch	Provides adapters for the Elasticsearch search and analytics engine.	2.1.1
com.ibm.streamsx.eventstore	Provides adapters for the Db2 Event Store database.	2.0.3
com.ibm.streamsx.hbase	Provides support for interacting with Apache HBase.	3.6.0
com.ibm.streamsx.hdfs	Provides operators that can read and write data from Hadoop Distributed File System (HDFS) version 2 or later.	4.4.1
com.ibm.streamsx.inet	Provides support for common internet protocol client functions and operators.	3.1.0
com.ibm.streamsx.iot	Provides Internet of Things (IoT) integration with IBM Watson IoT Platform.	1.2.0
com.ibm.streamsx.jdbc	Enables Streams applications to work with databases via JDBC.	1.6.0
com.ibm.streamsx.jms	Provides operators and functions that help you interact with JMS systems such as IBM MQ or Apache ActiveMQ.	1.1.0
com.ibm.streamsx.json	Provides standard transforms between SPL values and JSON objects.	1.5.2
com.ibm.streamsx.kafka	Enables Streams applications to easily integrate with Apache Kafka.	1.9.4
com.ibm.streamsx.mail	Provides support for the SMTP and IMAP protocols.	2.0.0
com.ibm.streamsx.messagehub	Enables Streams applications to work with IBM Event Streams.	1.9.3
com.ibm.streamsx.messaging	Includes operators and functions that help you interact with messaging systems such as Kafka, JMS, XMS, and MQTT.	5.4.1
com.ibm.streamsx.mqtt	Enables Streams applications to work with MQTT providers.	1.0.1
com.ibm.streamsx.network	Provides operators and functions for ingesting and parsing raw network data at the packet level.	3.2.1
com.ibm.streamsx.objectstorage	Provides the ability to read and write data from and to IBM Cloud Object Storage, respectively.	1.9.2
com.ibm.streamsx.rabbitmq	Provides operators that allows your Streams applications to read and send messages from RabbitMQ.	1.2.1
com.ibm.streamsx.sparkmllib	Allows Apache Spark’s MLlib library to be used for real-time scoring of data.	1.2.0
com.ibm.streamsx.topology	Provides mechanisms to build IBM Streams applications in other programming languages, with Java, Scala and Python support.	1.12.10
spl	Provides a set of operators, types, and functions that are specific to IBM Streams.	1.4.0