What's new in version 2.0.0

Kafka Toolkit > com.ibm.streamsx.kafka 2.2.1 > What's new and what has changed > What's new in version 2.0.0

New Features

The included Kafka client has been upgraded from version 2.1.1 to 2.2.1. Support for Kafka broker 2.2 has been added.

The toolkit has enhancements for the KafkaConsumer when it is used in an autonomous region (i.e. not part of a consistent region):

  • The KafkaConsumer operator can now participate in a consumer group with startPosition parameter values Beginning, End, and Time.
  • After re-launch of the PE, the KafkaConsumer operator does not overwrite the initial fetch offset to what the startPosition parameter is.

To achieve this new startPosition handling, the application must include a JobControlPlane operator when startPosition is not Default.

Incompatible changes, which may break existing applications

The behavior of the KafkaConsumer operator changes when

  1. the operator is not used in a consistent region, and
  2. the startPosition parameter is used with Beginning, End, Time, or Offset.

In all other cases the behavior of the KafkaConsumer is unchanged. Details of the changes, including sample code, can be found in the Toolkit documentation on Github.

Previous versions of the KafkaConsumer operator did not enable group management in autonomous regions, with startPosition different from Default, even when a group-ID was configured. Since toolkit version 2.0, group management is enabled with all startPosition values except Offset.

The conditions for a consumer operator to be part of a consumer group are now identical for consistent and autonomous region:

  1. A group identifier must be specified
  2. The operator must not have a control input port
  3. The partitions parameter must not be used

Correct handling of the initial start position requires always a JobControlPlane operator in the application graph, also when consistent region is not used. Without a JobControlPlane in the application graph, the Consumer operator will fail to initialize when startPosition is used and is not Default.

Since toolkit version 2.0, the startPosition has effect only when the consumer operator is initially launched at job submission. When the consumer's PE is re-launched or when new PEs are launched after changing the width of a parallel region, the fetch position is not reset to what startPosition is - in opposite to previous versions. More precisely, the fetch position is now only then reset for a topic partition, when the consumer has not yet committed offsets for the partition. In all other cases, the consumer continues fetching at the last committed offset.