Developing Java primitive operators

Java primitive operators use the Java Operator API to receive, inspect, and submit tuples.

They are seamlessly integrated into SPL. Java primitive operators have unique names. The SPL compiler can enforce compile-time checks of operator parameters, and on the number of input and output ports. Specialized operator models describe such constraints.

Java primitive operators also provide implementation encapsulation. A user of the operator does not need to know the name of the Java class and the required class path for the operator implementation. This information is described directly in the operator model. SPL developers that use a Java primitive operator do not need to know that the operator they are invoking is implemented in Java.

For more information, see Operators implemented in Java and Developing Java primitive operators.

Creating a Java primitive operator

Create a Java primitive operator that you can use in your streaming applications. See the Java primitive operator guide for a complete example.

Before you begin

Setting up VS Code for Java development

The Language Support for Java(TM) by Red Hat extension is required. When you install the IBM Streams extension, you will also be prompted to install this Java extension if it is not already installed.

You may also want to install the Java Extension Pack, which includes other extensions to help you test and debug your Java applications.

Java SE Development Kit (JDK) 11 or above is required for the Java extension(s) to run. For more information about installing a JDK, see Installing a Java Development Kit (JDK). Once JDK 11 is installed, configure VS Code to use the JDK by updating the java.home setting in VS Code’s User or Workspace settings. Example:

"java.home": "/Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home"

For more information, see Getting Started with Java in VS Code.

Configuring the Java runtime

Java SE Development Kit (JDK) 8 is required for developing Java primitive operators. For more information about installing a JDK, see Installing a Java Development Kit (JDK). Once JDK 8 is installed, configure VS Code to use the JDK by updating the java.configuration.runtimes setting in VS Code’s User or Workspace settings. Example:

"java.configuration.runtimes": [
  {
    "name": "JavaSE-1.8",
    "path": "/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home",
    "default": true
  }
]

For more information, see Configure JDK.

Procedure

1. Add your Streams 5.5 instance to the Streams Explorer in VS Code if you haven’t already.
2. Bring up the Command Palette and select Create Java Primitive Operator.
3. Follow the prompts to create the Java primitive operator. Specify the following:
   • Windows users only: Create an empty folder which will container the operator and set this folder to be the workspace.
   • Project folder path: The project folder path for the primitive operator. Use the Browse… button to create and select an empty folder on your machine. The primitive operator files will be created in this folder. (should be workspace for windows users)
   • Operator namespace: The SPL namespace for the primitive operator.
   • Operator name: The name of the primitive operator.
   • Operator processing pattern: The operator processing pattern identifies whether the operator has input ports, output ports, or both and the type of tuple flow the operator provides.

4. Add the Log4j 1.2 dependency.

   1. Download the JAR file from one of the mirrors here.
   2. Unpackage the .tar.gz or .zip file.
   3. Designate a folder on your machine where Java JAR dependencies will be stored (e.g., /Users/someuser/Documents/javaJars). Locate the log4j-1.2.XX.jar file inside the unpackaged folder and move or copy it to the designated folder.

   4. Update the java.project.referencedLibraries setting in VS Code’s User or Workspace settings to add the folder containing the JAR file. The path should be absolute and can contain a glob pattern. Example:

      "java.project.referencedLibraries": [
        "/Users/someuser/Documents/javaJars/*.jar"
      ]
      

      If your operator depends on other .jar files, add them to the same folder to bring in these dependencies.

Results

Based on the selected operator processing pattern, a Java primitive operator is created in the specified project folder and the operator’s Java source file (<operator-name>.java) is opened for editing.

The folder structure for the Java project is:

/+ <project-folder>
   /+ impl
      /+ java
         /+ bin
         /+ src
            /+ <operator-namespace>
               /+ <operator-name>.java
   /+ Makefile

Note that a Makefile file is also created. This is used to build the Java primitive operator. The file looks like:

CLASS_PATH=$(STREAMS_INSTALL)/lib/com.ibm.streams.operator.jar:$(STREAMS_INSTALL)/lib/com.ibm.streams.operator.samples.jar
DEST_DIR=impl/java/bin
SOURCE_FILE=impl/java/src/<operator-namespace>/<operator-name>.java
TOOLKIT_NAME=<project-folder>

all: compile-java build-toolkit

compile-java:
	javac -cp $(CLASS_PATH) -d $(DEST_DIR) $(SOURCE_FILE)

build-toolkit:
	$(STREAMS_INSTALL)/bin/spl-make-toolkit -i . -n $(TOOLKIT_NAME)

What to do next

Implement the primitive operator using the Java Operator API. This depends on the operator processing pattern you chose.

• If your operator consumes incoming tuples and produces outgoing tuples, see Implementing an operator using the Java Operator API.
• If your operator produces outgoing tuples as a source operator, see Implementing a source operator with the Java Operator API.
• If your operator consumes incoming tuples as a sink operator, see Implementing a sink operator using the Java Operator API.

Use the TODO comments in the source file as a guide.

For more information, see Developing Java primitive operators and the Java Operator Development Guide.

Adding the src folder to the Java source path

If you see a message like the following in the PROBLEMS panel after opening the .java source file, then you’ll need to add the src folder to the Java source path.

<operator-name>.java is not on the classpath of project <project_name>, only syntax errors are reported

Right-click on the src folder and select Add Folder to Java Source Path.

Building the operator

When you’re ready, build the Java primitive operator to generate a toolkit that you can use in your streaming applications.

Building a Java primitive operator

Build a Java primitive operator to generate a toolkit that you can use in your streaming applications. See the Java primitive operator guide for a complete example.

Before you begin

Your Java project must have the following folder structure.

Note: If you created your operator using the Create Java Primitive Operator command, then your project will have been created with the correct structure.

/+ <project-folder>
   /+ impl
      /+ java
         /+ bin
         /+ src
            /+ <operator-namespace>
               /+ <operator-name>.java
   /+ Makefile

The Makefile must compile the Java source file(s) using javac and build a toolkit using spl-make-toolkit. A simple Makefile would be the following:

CLASS_PATH=$(STREAMS_INSTALL)/lib/com.ibm.streams.operator.jar:$(STREAMS_INSTALL)/lib/com.ibm.streams.operator.samples.jar
DEST_DIR=impl/java/bin
SOURCE_FILE=impl/java/src/<operator-namespace>/<operator-name>.java
TOOLKIT_NAME=<project-folder>

all: compile-java build-toolkit

compile-java:
	javac -cp $(CLASS_PATH) -d $(DEST_DIR) $(SOURCE_FILE)

build-toolkit:
	$(STREAMS_INSTALL)/bin/spl-make-toolkit -i . -n $(TOOLKIT_NAME)

Note: If you have created multiple operators in the same project, you must update the Makefile to compile all of the .java source files. Example:

SOURCE_FILE=impl/java/src/<operator-namespace-1>/*.java impl/java/src/<operator-namespace-2>/*.java

Procedure

1. Add your Streams 5.5 instance to the Streams Explorer in VS Code if you haven’t already.
2. Right-click on a Java project folder or .java file containing a primitive operator and select Build Java Primitive Operator.
3. Follow the prompts to submit the build.

Results

After a build completes successfully, the indexed toolkit is extracted to the project folder (overwriting the existing files). You will be presented with two options:

• Download Toolkit: Downloads the toolkit archive file (.tgz) to the toolkit folder.
• Add Toolkit to Toolkit Path: Adds the toolkit to a toolkit path folder (defined in the Toolkit Paths setting). This makes the indexed toolkit available to use in your Streams applications.

The toolkit will have the following folder structure. The authored files are marked with + and generated files are marked with *.

/+ <project-folder>
   /+ impl
      /+ java
         /+ bin
            /* <operator-namespace>
               /* <operator-name>.class
               /* <operator-name>$StreamsModel.java
               /* <operator-name>$StreamsModel.class
         /+ src
            /+ <operator-namespace>
               /+ <operator-name>.java
   /* <operator-namespace>
      /* <operator-name>
         /* <operator-name>.xml
   /+ Makefile
   /* toolkit.xml

What to do next

Learn how to use the toolkit in your SPL applications here.