Toolkit Usage Overview

Edit me

Introduction

The Message Hub toolkit contains two operators, the MessageHubConsumer, and the MessageHubProducer. The MessageHubConsumer operator consumes messages from an Event Streams topic and creates Tuples which are processed by other downstream operators of the Streams application. It is a source operator within a Streams application.

The MessageHubProducer operator creates Kafka messages in the Event Streams service from tuples and acts therefore as a sink operator within your Streams application. The MessageHubConsumer has a one-to-one relationship between Kafka messages and produced tuples. for the MessageHubProducer there is a one-to-N relationship between tuples and Event Streams messages, where N is the number of topics for which a tuple is produced. The Event Streams cloud service is based on Kafka, so that this toolkit and most descriptions are strongly related to those of the Kafka toolkit. Read more about how to use these operators in the SPL documentation.

Kafka client versions

The Message Hub toolkit contains the Java Kafka clients package. Which toolkit version ships with which kafka-clients version, can be found in the following table.

toolkit version streamsx.kafka base version kafka-clients version
3.3.x 3.2.x 2.5.1
3.2.2 3.1.3 2.3.1
3.2.0 - 3.2.1 3.1.0 2.3.1
3.1.0 3.0.4 2.3.1
3.0.x 3.0.x 2.3.1
2.1.x - 2.2.x 2.1.x - 2.2.x 2.2.1
2.0.2 2.0.1 2.2.1
2.0.1 2.0.0 2.2.1
2.0.0 2.0.0 2.2.1
1.9.4 1.9.5 2.1.1
1.9.3 1.9.4 2.1.1
1.9.2 1.9.3 2.1.1
1.9.0 - 1.9.1 1.9.0 - 1.9.1 2.1.1
1.8.0 1.8.0 1.0.0
1.7.4 1.7.3 1.0.0
1.7.0 - 1.7.3 1.7.0 - 1.7.3 1.0.0
1.6.x 1.6.x 1.0.0
1.5.x 1.5.x 1.0.0
1.4.x 1.4.x 1.0.0
1.3.x 1.3.x 1.0.0
1.1.0 - 1.2.x 1.1.0 - 1.2.x 0.10.2.1
1.0.x 1.0.0 0.10.2

Common consumer patterns and use cases

Use of consistent region

Kafka, as the base of the Event Streams cloud service, itself has the capability of at-least-once delivery from producers to consumers. To keep this delivery semantics within Streams applications consuming messages from Kafka topics, it is recommended to consider using a consistent region within the Streams application unless used operators do not support consistent region.

Overview

Assumptions:

  • One consumer operator consumes messages from one single topic with a string message, for example a JSON message
  • For a production environment, the consumer starts consuming at the default start position
  • Event Streams guarantees no ordering of messages accross partitions.
  • Credentials for the Event Streams service instance is stored in an application configuration.

There are three standard patterns for Streams reading messages from Event Streams.

  • All partitions - A single MessageHubConsumer invocation consumes all messages from all partitions of a topic
  • Kafka consumer group - the partitions of a topic are automatically assigned to multiple MessageHubConsumer invocations for consumption
  • Assigned partitions - Multiple MessageHubConsumer invocations with each invocation assigned specific partitions.

The MessageHubConsumer operator can be configured with additional Kafka consumer properties. These can be specified in a property file or in the application configuration that contains also the credentials. The examples in the standard patterns use an application configuration. Some operator parameters, like groupId, and clientId map directly to properties. Other properties are adjusted by the operator. Which one, can be reviewed in the SPL documentation of the operators.

Property example of the application configuration for a consumer

property name property value
messagehub.creds { “api_key”: “Tv39…eT”, …, …, “user”: “token” }
max.poll.records 2000
partition.assignment.strategy org.apache.kafka.clients.consumer.CooperativeStickyAssignor

Each such a collection of properties has a name, the name of the application configuration, which must be configured as the appConfigName parameter.

Samples

It is also worth looking at the samples of the Kafka toolkit, which can be found here.

Updated: