What's new in version 1.8.0

Kafka Toolkit > com.ibm.streamsx.kafka 2.2.1 > What's new and what has changed > What's new in version 1.8.0

  • The KafkaConsumer can subscribe dynamically to multiple topics by specifying a regular expression for the new pattern parameter. Assignment of partitions happens for matching topics at the time of periodic check. When someone creates a new topic with a name that matches, a rebalance will happen almost immediately and the consumers will start consuming from the new topic.

  • The control port of the KafkaConsumer allows assignment with default fetch position. New SPL functions to generate the JSON string for the control port have been added.

  • KafkaConsumer: Offsets can be committed after a time period, not only when a tuple count is reached. The new commitPeriod operator parameter lets you specify a time period in seconds for committing offsets of submitted tuples.

  • KafkaConsumer: The time policy for offset commit is now the default policy when not in consistent region. The time policy avoids too high commit request rates, which can occur with count based policy and high tuple rates. The default commit interval is 5 seconds.

  • The operators can now be configured with a config checkpoint clause when used in an autonomous region. The KafkaProducer operator simply ignores the config instead of throwing an error at compile time. The KafkaConsumer operator can be configured with operator driven and periodic checkpointing. Checkpointing is in effect when the operator is configured with the optional input port. Then, the operator checkpoints or restores the assigned partitions and resumes fetching at last committed offset.