Ingest Data from a Source into Snowflake

This example pipeline demonstrates how to use the Snowflake - Snowpipe Streaming Snap to ingest data into the Snowflake database from a source as and when it is available.

Here's the execution flow of the pipeline:

  • Data generation: The Sequence Snap generates a set number of data records.

  • Data mapping: The first Mapper Snap maps each generated value to the num field.

  • Data Streaming: The Snowflake Snowpipe Streaming Snap ingests the JSON-formatted data into a specified Snowflake table as soon as the data is available.



Download this Pipeline

  1. Configure the Sequence Snap to generate ten documents.
  2. Configure the Mapper Snap to pass the values to the downstream Snowflake- Snowpipe Streaming Snap.


  3. Configure the Snowflake - Execute Snap to create or replace an existing table with the column datatype.
  4. Configure the Mapper Snap to pass the values to the downstream Snowflake- Snowpipe Streaming Snap.


  5. Configure the Snowflake - Snowpipe Streaming Snap to ingest the data into the Snowflake database as and when available.


  6. Validate the Snap. On validation, the data is successfully inserted into the Snowflake database.