S3 Download

Overview

You can use this Snap to download S3 objects from the S3 bucket.



Limitations

The current Snap functionality supports the AWS S3 Cloud Service and is applicable for the AWSGovCloud setup.

Snap views

View Description Examples of upstream and downstream Snaps
Input An upstream Snap is optional. Any document with key-value pairs to evaluate expression properties. Each input document, if any, results in one download operation of the Snap.
Output Binary data downloaded from AWS S3 is specified in the Bucket and Object Key fields with header information about the binary stream.
Error

Error handling is a generic way to handle errors without losing data or failing the Snap execution. You can handle the errors that the Snap might encounter when running the pipeline by choosing one of the following options from the When errors occur list under the Views tab. The available options are:

  • Stop Pipeline Execution Stops the current pipeline execution when an error occurs.
  • Discard Error Data and Continue Ignores the error, discards that record, and continues with the remaining records.
  • Route Error Data to Error View Routes the error data to an error view without stopping the Snap execution.

Learn more about Error handling in Pipelines.

Snap settings

Legend:
  • Expression icon (): Allows using pipeline parameters to set field values dynamically (if enabled). SnapLogic Expressions are not supported. If disabled, you can provide a static value.
  • SnapGPT (): Generates SnapLogic Expressions based on natural language using SnapGPT. Learn more.
  • Suggestion icon (): Populates a list of values dynamically based on your Snap configuration. You can select only one attribute at a time using the icon. Type into the field if it supports a comma-separated list of values.
  • Upload : Uploads files. Learn more.
Learn more about the icons in the Snap settings dialog.
Field / Field set Type Description
Label String

Required. Specify a unique name for the Snap. Modify this to be more appropriate, especially if more than one of the same Snaps is in the pipeline.

Default value: S3 Download

Example: S3 Download
Bucket String/Expression/ Suggestion

Required. Specify the S3 bucket name, from where an S3 object is to be downloaded.

Warning: Do not add S3:/// before bucket name, because the Snap can fail.

Bucket names are unique globally and can be accessed without the region name in most cases. If you cannot access a bucket name without its region name, you can specify the region information with the following syntax:

<S3_bucket_name>@<region_name>

Note:
If you enter an incorrect region name, but the bucket name is valid, the AWS S3 service might successfully access the bucket without errors.

  • You can access an S3 bucket in an S3 Virtual Private Cloud (VPC) endpoint by specifying the bucket name with the following syntax:

    • <S3_bucket_name>@<VPC_S3_endpoint>

  • You can access an S3 Express One Zone bucket with the following syntax:

    • <bucket-name>--<region>-<available-zone>--x-s3

  • S3 Express One Zone does not support the following bucket name pattern:

    • <bucket>@<region_info>.

Default value: None.

Examples:
Object Key String/Expression/ Suggestion

Required. Specify the S3 object key name, which may include one or more forward-slash ('/') characters.

Note: The forward-slash character is part of the S3 object key name, and there is no folder object defined in AWS S3. The Snap uses the existing Object Key value as a prefix to produce the suggested list. The maximum length of the suggested list is 1,000.

Default value: None.

Examples:

  • test.csv

  • abc/test.json

  • abc/xyz/test.xml
Show Advanced Properties Checkbox Select this checkbox to display the advanced properties.

Deselect this checkbox to hide the advanced properties.

Default status: Deselected
Thread Pool Size Integer/Expression

Appears when you select Show Advanced Properties checkbox.

Specify the maximum number of threads to use to download multiple S3 objects in parallel.

Default value: 10

Example: 5
Maximum Retries Integer/Expression Appears on selecting Show Advanced Properties checkbox.

Required. Specify the maximum number of retry attempts to perform in case of a temporary network loss.

Default value: 3

Example: 5
Additional output headers Use this field set to define key-value pairs as additional headers in the binary output.
Note: If the downstream Snap is HTTP Client Snap, these headers are used to evaluate expressions and perform the multipart file upload from various file protocols multiple times.
Key String Specify the key name for the additional output header.
Value String/Expression Specify the value for the additional output header.
Version ID String/Expression/ Suggestion Appears when you select Show Advanced Properties checkbox.

Specify or select the version ID of the S3 file object. If you leave this field empty, the Snap downloads the latest version. S3 versioning is not supported in S3 Express One Zone.

Note: You can use the Suggestion icon to view the list of version IDs for the S3 object in the Object Key field. Each line in the suggested list also includes the last modified date and the file size to help you select a version. If the versioning is not enabled in the S3 bucket, no version ID is suggested.

When you specify a static value in this field, you must enter only the version ID. The Snap ignores the last modified date and size information of a version when it downloads the version.

The versions for the following cases are ignored in the suggested list because you cannot download them:

  • If an S3 object had existed before enabling the versioning; therefore, its version does not have any version ID assigned to it.

  • Any version ID with a 'Deleted Marker' resource type.

Default value: N/A

Example: xvcnB8gPi37l3hbOzlsRFxjVwQ.numQz
Version ID Suggestion Interval Use this field set to configure the time interval for the version ID suggestion.
Note:

  • Enter two separate rows to enter a start date and an end date. If you provide only one row, the suggestion interval is considered from the specified data until the current date.

  • If you leave this field empty, the Snap suggests all version IDs. This may be helpful when a specific S3 file has many versions. This property is used only for the Version ID suggestion, not during the Snap preview or execution.
Year Integer Appears when you select Show Advanced Properties checkbox.

Enter the year as a 4-digit integer.
Month Integer Appears when you select Show Advanced Properties checkbox.

Enter the month as an integer.
Date Integer Appears when you select Show Advanced Properties checkbox.

Enter the day of the month.
Zone String/Suggestion Appears on selecting Show Advanced Properties checkbox.
Enter or select a time zone ID from the suggested list. For the UTC time zone, this field may be empty.
Warning: Only zone IDs are supported in the suggested list.
Get Object Tags Checkbox Appears when you select Show Advanced Properties checkbox.

Select this property to include object tags in the header of the output binary data. Learn more about object tags.

Note:

  • You must have the S3:GetObjectTagging permission to be able to use this feature.

  • S3 object tagging is not supported in S3 Express One Zone.

Default status: Deselected
Enable Staging Checkbox Appears when you select Show Advanced Properties checkbox.
Select this checkbox to download the entire file content into a local temporary file. When the download is completed, the Snap streams the data from the temporary file to the output view.
Note: Some downstream Snaps may take a long time to process large amounts of data, leading to the connection reset error, which causes the pipeline to fail. Therefore, to avoid such errors, we recommend you select this checkbox to save the data on your local disk.

Default status: Deselected
Snap execution Dropdown list
Choose one of the three modes in which the Snap executes. Available options are:
  • Validate & Execute: Performs limited execution of the Snap and generates a data preview during pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during pipeline runtime.
  • Execute only: Performs full execution of the Snap during pipeline execution without generating preview data.
  • Disabled: Disables the Snap and all Snaps that are downstream from it.

Default value: Execute only

Example: Validate & Execute

Examples