Amazon S3

You can create a data pipeline that loads CSV or JSON files from S3 storage to a cloud data warehouse. For each file, SnapLogic® AutoSync creates a table in the target. If a source file has a schema, AutoSync replicates that schema in the target. If a source file does not have a schema, AutoSync creates columns with a string type supported by the target, for example, for Snowflake, VARCHAR. Learn more about how AutoSync loads files.

To provide the information SnapLogic® AutoSync needs to connect to S3, supply new credentials in the wizard or select the saved credentials. The account AutoSync uses must have read permissions for the file storage location. The create credentials page lists the information necessary to create credentials in the AutoSync wizard.

Supported JSON data types

AutoSync supports the following data types for JSON files:

  • boolean
  • float
  • integer
  • object
  • string
  • varchar
  • number (38,0)

With JSON files, AutoSync stores numbers as numerical data types. With CSV files, all data is stored as strings. For example, 2.000 is stored as the number 2 with JSON, but with CSV, it is stored as a string.

Connection configuration

S3 properties include the following:

  • Account Properties:
    • Credential label: A unique, meaningful name such as S3-Shared-Sales. If a configuration with the same name exists, AutoSync displays an Asset conflict error message.
    • Access-key ID: The S3 Access key ID for authentication. For example, NAVRGGRV7EDCFVLKJH.
    • Secret Key: The S3 Secret key for authentication. For example, 2RGiLmL6bCujkKLaRuUJHY9uSDEjNYr+ozHRtg.
    • Share: (Optional) Select a user group to share this configuration with. Org admins create user groups to share credentials. If you are a member of a user group, you can select it from the dropdown. You can also select the global shared folder, which shares the credentials with everyone in your Org.
  • Validate and Save: After saving and validating, AutoSync adds the configuration to the list of saved credentials.
  • File type: After selecting your configuration, choose the file type, CSV or JSON.
  • Folder path: Enter the path to the source files. Bucket and folder names are case sensitive. You can only select files from existing buckets and folders. The file path must be entered as one of the following formats:
    • autosync-staging/testdata/csv or autosync-staging/testdata/csv/
    • s3:///autosync-staging/testdata/csv or s3:///autosync-staging/testdata/csv/
  • Select files to synchronize: Choose one or more source files to load and synchronize in the destination.