Consume Messages from a Kafka Topic

This example demonstrates how messages are read from a Kafka topic, and how the selected output documents are sent to the downstream Mapper view.


Kafka Consumer pipeline overview

Download this pipeline
  1. Configure the Kafka Consumer Snap. First, we configure the Kafka Consumer Snap to read the messages from the topic Sample_Topic under the Group ID consumergroup. We select the Output Mode as One output document per batch, because we want to view the output records in batches.

    Kafka Consumer Snap configuration

  2. Validate the Consumer output. After validation, we can view the output preview of the Kafka Consumer Snap as follows:

    Kafka Consumer Snap output preview


  3. Mapper Snap configuration showing mapping root set to batch
  4. Configure the Mapper Snap. We use the Mapper Snap to display only those fields that we require in the output view. Note that we set the Mapping root to $batch[*] to map the fields of each record in the batch. Hence, we configure the Mapper Snap as shown below:

    Mapper Snap configuration showing mapping root set to batch

  5. View the Mapper output. Upon successful execution, we see the output preview of the Mapper Snap as follows:

    Mapper Snap output showing selected batch fields

  1. Ensure the Kafka Consumer Snap is connected to the Mapper Snap output view.
  2. Verify that the Mapping root is correctly set to $batch[*] in the Mapper Snap.
  3. Validate that the Kafka Consumer produces batched output as expected.