Advanced Use Case (PolyBase Bulk Load)

This example demonstrates a broader enterprise ETL pipeline that combines the PolyBase Bulk Load Snap with additional data transformation steps. The pipeline reads source data, applies transformations, and bulk-loads the results into a SQL Server table using PolyBase, showcasing how the Snap fits into a multi-stage data pipeline.

Download this Pipeline
  1. Extract: Configure an upstream data source Snap to read records from the source system.
  2. Transform: Use a Mapper Snap to reshape or enrich the source data to match the target table schema. Apply any required filtering or aggregation using additional Transform Snaps.
  3. Load: Configure the PolyBase Bulk Load Snap with the appropriate Table Name, Schema Name, and Bulk insert mode (Append or Overwrite). Enable Create table if not present if the target table may not yet exist.
    Note: Microsoft PolyBase does not support varchar entries containing more than 1000 characters. Use the Azure SQL - Bulk Load Snap as an alternative for such data.
  4. Execute the pipeline. The Snap stages the transformed data as CSV in Azure Blob storage before loading it into the target table using SQL Server PolyBase commands.
To reuse the example pipelines:
  1. Download and import the SLP file into your Environment.
  2. Configure Snap accounts.
  3. Provide Pipeline parameters, if any.