Load Data from Salesforce to Redshift via SQS Messaging

The following describes a pipeline, with a broader business logic involving multiple ETL transformations, that shows how typically in an enterprise environment, the SQS messaging functionality is used. The pipeline download is available in the Downloads section.

The pipeline reads and moves documents from the Salesforce instance to the SQS Messaging and then to the Redshift Database.

The ETL transformations:

Extract: The Salesforce Read Snap extracts the Records to be sent to the SQS Messaging instance.

Transform: The JSON Formatter Snap transform the records into the JSON format.

Load: The SQS Producer Snap receives the records from the JSON input.

Extract: The SQS Consumer Snap reads the messages from the specified Queue.

Transform: The Binary to Document Snap converts the records into Document format to be moved onto the Redshift instance.

Load: The Redshift Bulk Load Snap loads the documents into the Redshift database.


Amazon Example Pipeline

Download this pipeline
The Snap is configured with the following parameters:

To successfully reuse pipelines:
  1. Download and import the pipeline in to the SnapLogic Platform.
  2. Configure Snap accounts, as applicable.
  3. Provide pipeline parameters, as applicable.