Snowflake CDC

Overview

You can use this Snap to get the inserted, updated, and deleted data from a defined Snowflake table using Snowflake Streams.


Snowflake CDC Overview

Features

  • Transaction retrieval: This Snap retrieves transactions executed on a table. It can handle Insert, Update, and Delete transactions.

Prerequisites

  • A valid Snowflake account with the required permissions to access Snowflake Streams.
  • Snowflake Stream must be created on the target table.
  • Appropriate privileges to read from the Stream and underlying table.

Snap views

Type Description Examples of upstream and downstream Snaps
Input This Snap supports at the most one document input view. Mapper
Output The output is a document with all the statement data. Mapper
Learn more about Error handling.

Snap settings

Note: Learn about the common controls in the Snap settings dialog.
Field/Field set Description

Label

String

 Required.Specify a unique name for the Snap. Modify this to be more appropriate, especially if there are more than one of the same Snap in the pipeline.

Default value: Snowflake CDC

Example: SnowflakeDB CDC
Stream name

String/Expression

 Required. Specify the name of the stream where the changes are logged.

Default value: None

Example: HR
Polling interval

Integer

 The interval at which the Snap checks for changes. This value represents the frequency of data polling.

Default value: None

Example: 5
Polling unit

Dropdown list

 Choose the unit of time for the polling interval. The available options are:
  • SECONDS
  • MINUTES
  • HOURS

Default value: None

Example: MINUTES
Batch output

Checkbox

 Select this checkbox to display the data in bulk within the content object in the output.

Default status: Deselected
Batch output size

Integer

 Appears only when you select the Batch output checkbox. Specify the number of documents to process in a single batch.

Default value: None

Example: 100

Snap execution

Dropdown list
Choose one of the three modes in which the Snap executes. Available options are:
  • Validate & Execute: Performs limited execution of the Snap and generates a data preview during pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during pipeline runtime.
  • Execute only: Performs full execution of the Snap during pipeline execution without generating preview data.
  • Disabled: Disables the Snap and all Snaps that are downstream from it.

Default value: Execute only

Troubleshooting

Error Reason Resolution
No data in the output There are no changes in the defined period or the Stream is empty. Verify the specified time range to ensure that it encompasses periods with changes. Also verify that the Snowflake Stream exists and has captured changes.
Stream not found The specified stream does not exist in the schema. Verify that the Snowflake Stream has been created on the source table before running the pipeline.
Insufficient privileges The role does not have the required permissions to access the stream or tables. Grant the account role access to the database, schema, tables, and streams. Ensure the role has SELECT privileges on the stream.

Examples