Publish and consume messages in Kafka using Schema Registry authentication

This pipeline demonstrates a Kafka message flow using Schema Registry authentication. It creates sample user data, sends it to a Kafka topic using Avro serialization, extracts metadata information, and then consumes the message from the same topic to retrieve the original data along with Kafka metadata.
Kafka Message flow pipeline

Download this pipeline

Pipeline summary:

  • JSON Generator: Generates sample JSON content containing user information with firstName "vim", lastName "snap", and age 1.
  • Kafka Producer: Publishes the JSON data to test-topic with Avro serialization for both key and value, using leader acknowledgement.
  • Mapper: Extracts and maps Kafka message metadata including offset, topic, and partition information from the producer output.
  • Kafka Consumer: Consumes messages from the Kafka topic using the metadata captured from the producer, seeking to the specific offset and partition with string deserialization.
  • Mapper: Extracts and maps the consumed message data including key, value, and topic information from the Kafka consumer output.
Configure the Kafka OAuth2 Account as follows:
Kafka OAuth2 account configuration

On validation, the account validates successfully.