Kafka Acknowledge

Overview

You can use the Kafka Acknowledge Snap to notify the Kafka Consumer Snap to commit an offset at the specified metadata in each input document.
Note:
  • This Snap should be used only if the Auto commit field in the Consumer Snap is not selected (set to false).
  • This Snap no longer requires a Kafka account.

Kafka Acknowledge Snap Configuration

Prerequisites

  • A Confluent Kafka server.
  • An upstream Snap (for example, a Kafka Consumer Snap) that sends metadata to the Kafka Acknowledge Snap in a Pipeline.

Snap views

Type Description Examples of upstream and downstream Snaps
Input Metadata from an upstream Kafka Consumer Snap. The input data schema is as follows:
{
                                    "metadata": {
                                    "topic": "xyz",
                                    "partition": 2,
                                    "offset": 523,
                                    "consumer_group": "CopyGroup1",
                                    "client_id": "17a9bbc7-da8f-45f8-813e-1ebca9b80383",
                                    "tracker_index": 0,
                                    "batch_size": 500,
                                    "batch_index": 1,
                                    "record_index": 23,
                                    "auto_commit": false
                                    }
                                    }
Output Kafka messages that have been processed and acknowledged.

If the Auto Commit field is set to false in the input document and a notification is sent successfully to the corresponding Consumer Snap, the output schema is similar to the following:

{
                                    "status": "success",
                                    "original": {
                                    "metadata": {
                                    "consumer_group": "abc",
                                    "topic": "xyz",
                                    "partition": 123,
                                    "offset": 456,
                                    "auto_commit": false
                                    }
                                    }
                                    }

If the Auto commit field is set to true in the input document, the output schema is similar to the following:

{
                                    "status": "Auto-commit is on",
                                    "original": {
                                    "metadata": {
                                    "consumer_group": "abc",
                                    "topic": "xyz",
                                    "partition": 123,
                                    "offset": 456,
                                    "auto_commit": true
                                    }
                                    }
                                    }
Learn more about Error handling.

Snap settings

Legend:
  • Expression icon (): Allows using pipeline parameters to set field values dynamically (if enabled). SnapLogic Expressions are not supported. If disabled, you can provide a static value.
  • SnapGPT (): Generates SnapLogic Expressions based on natural language using SnapGPT. Learn more.
  • Suggestion icon (): Populates a list of values dynamically based on your Snap configuration. You can select only one attribute at a time using the icon. Type into the field if it supports a comma-separated list of values.
  • Upload : Uploads files. Learn more.
Learn more about the icons in the Snap settings dialog.
Field/Field set Type Description
Label String

Required. Specify a unique name for the Snap. Modify this to be more appropriate, especially if more than one of the same Snaps is in the pipeline.

Default value: Kafka Acknowledge

Example: Kafka_Acknowledge
Metadata path String Required. Specify the JSON path of the metadata within each input document.

Default value: metadata

Example: $metadata
Snap execution Dropdown list

Select one of the three modes in which the Snap executes:

  • Validate & Execute: Performs limited execution of the Snap and generates a data preview during Pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during Pipeline runtime.
  • Execute only: Performs full execution of the Snap during Pipeline execution without generating preview data.
  • Disabled: Disables the Snap and all Snaps downstream from it.

Default value: Validate & Execute

Example: Validate & Execute
Acknowledging Messages