Pipeline: Multi-file reader with S3

Read multiple files from S3 and write it to a single file back in S3

Requirements

Snaps used
The following Snaps are used:
  • Multi File Reader
  • File Writer

Other requirements

  • An S3 account
  • Several files within an S3 bucket with similar names. This example will utilize the wildcard functionality of the Multi File Reader Snap.

Configuration

  1. Within Manager, create an account for your S3 credentials.
  2. Within Designer, create a new pipeline.
  3. Add and configure a Multi File Reader Snap.
    1. Set Folder/File to the bucket path within S3, such as s3:///demo-sl-bucket/
    2. Set Wildcard to a value that will pull the files you need, such as delta.csv-*
  4. Add and configure a File Writer Snap.
    • Set the File Name to the name of the file where you want the results written to, such as s3:///demo-sl-write-/fulldelta.csv