Amazon S3 as a source

Select the S3 endpoint to load CSV or JSON files from Amazon S3 storage to a cloud data warehouse. For each file, SnapLogic AutoSync creates a table with the same name in the target.

If a source file has a schema, AutoSync replicates that schema in the target. If a source file doesn't have a schema, AutoSync creates columns with a string type supported by the target (for example, for Snowflake, VARCHAR).

AutoSync stores CSV values in the target as strings, unless you enable Infer data types for CSV files in the connection configuration. To infer column types, AutoSync checks up to the first 100 rows and can detect the following types:
  • STRING
  • NUMBER
  • BOOLEAN
  • TIMESTAMP
With type inference enabled, AutoSync treats empty strings as null values.

Learn more about how AutoSync loads files.

To provide the information SnapLogic AutoSync needs to connect to S3, supply new credentials in the wizard or select saved credentials. The account must have read permissions for the file storage location.

Supported JSON data types

AutoSync supports the following data types for JSON files:

  • boolean
  • float
  • integer
  • object
  • string
  • varchar
  • number (38,0)

With JSON files, AutoSync stores numbers in the corresponding numeric type in the target.

Supported Snap account types

Designer and Classic Manager provide multiple Snap account types for most endpoints and not all types are compatible with AutoSync. When you create or edit a data pipeline, the existing credentials list includes only compatible accounts.

For S3, AutoSync supports:

  • Amazon S3 Account

Connection configuration

When you create S3 credentials in AutoSync, the properties include :

  • Account Properties:
    • Credential label: A unique, meaningful name such as S3-Shared-Sales. If a configuration with the same name exists, AutoSync displays an Asset conflict error message.
    • Access-key ID: The S3 Access key ID for authentication. For example, NAVRGGRV7EDCFVLKJH.
    • Secret Key: The S3 Secret key for authentication. For example, NXXxXxXNxXxxxXXxXxXXXXNxXXXxXXx+xxXXxx.
    • Share: (Optional) Select a user group to share this configuration with. Org admins create user groups to share credentials. If you are a member of a user group, you can select it from the dropdown. You can also select the global shared folder, which shares the credentials with everyone in your Org.
  • Validate and save: After saving, AutoSync adds the configuration to the list of saved credentials.
  • File type: After selecting your configuration, choose the file type, CSV or JSON.
  • Folder path: Enter the path to the source files. Bucket and folder names are case sensitive. You can only select files from existing buckets and folders. Enter the file path as one of the following formats:
    • autosync-staging/testdata/csv or autosync-staging/testdata/csv/
    • s3:///autosync-staging/testdata/csv or s3:///autosync-staging/testdata/csv/
  • Select files to synchronize: Choose one or more source files to load and synchronize in the destination.
  • Infer data types for CSV files: Only shows when you select the CSV file type. When enabled, AutoSync infers data types from up to the first 100 rows of each file.