S3 Upload

Overview

You can use this Snap to upload binary data to S3 objects.

Note:

You can upload multiple S3 objects in multi-threads by using expression enabler for the Bucket and Object Key. If the input binary data is larger than Multipart Size (MB), it performs the write operation by using the Multipart Upload API in multiple threads.



Snap views

View Description Examples of upstream and downstream Snaps
Input The Input view receives binary data containing the formatted JSON content from the upstream JSON Formatter transformation, providing the file content that needs to be uploaded to the S3 bucket.
Output The Output view produces document data containing the upload operation status and metadata, confirming successful storage of the file in the specified S3 bucket location.
Error

Error handling is a generic way to handle errors without losing data or failing the Snap execution. You can handle the errors that the Snap might encounter when running the pipeline by choosing one of the following options from the When errors occur list under the Views tab. The available options are:

  • Stop Pipeline Execution Stops the current pipeline execution when an error occurs.
  • Discard Error Data and Continue Ignores the error, discards that record, and continues with the remaining records.
  • Route Error Data to Error View Routes the error data to an error view without stopping the Snap execution.

Learn more about Error handling in Pipelines.

Snap settings

Legend:
  • Expression icon (): Allows using pipeline parameters to set field values dynamically (if enabled). SnapLogic Expressions are not supported. If disabled, you can provide a static value.
  • SnapGPT (): Generates SnapLogic Expressions based on natural language using SnapGPT. Learn more.
  • Suggestion icon (): Populates a list of values dynamically based on your Snap configuration. You can select only one attribute at a time using the icon. Type into the field if it supports a comma-separated list of values.
  • Upload : Uploads files. Learn more.
Learn more about the icons in the Snap settings dialog.
Field / Field set Type Description
Label String

Required. Specify a unique name for the Snap. Modify this to be more appropriate, especially if more than one of the same Snaps is in the pipeline.

Default value: S3 Upload

Example: S3 Upload
Bucket String/Expression/ Suggestion

Required. Specify the S3 bucket name where the S3 object to be downloaded.

Warning: Do not add S3:/// before bucket name, because the Snap can fail.

Bucket names are unique globally and can be accessed without the region name in most cases. If you cannot access a bucket name without its region name, you can specify the region information with the following syntax:

<S3_bucket_name>@<region_name>

Note:

If you enter an incorrect region name, but the bucket name is valid, the AWS S3 service might successfully access the bucket without errors.

  • You can access an S3 bucket in an S3 Virtual Private Cloud (VPC) endpoint by specifying the bucket name with the following syntax:

    • <S3_bucket_name>@<VPC_S3_endpoint>

  • You can access an S3 Express One Zone bucket with the following syntax:

    • <bucket-name>--<region>-<available-zone>--x-s3

  • S3 Express One Zone does not support the following bucket name pattern:

    • <bucket>@<region_info>.

  • S3 Express One Zone does not support VPC.

Default value: N/A

Examples:
Object Key String/Expression/ Suggestion

Required. Specify the S3 object key name, which may include one or more forward-slash ('/') characters.

Note:

The forward-slash character is part of the S3 object key name, and there is no folder object defined in AWS S3. The Snap uses the existing Object Key value as a prefix to produce the suggested list. The maximum length of the suggested list is 1,000.

Default value: N/A

Examples:

  • test.csv

  • abc/test.json

  • abc/xyz/test.xml
Show Advanced Properties Checkbox

Select this checkbox to display the advanced properties.

Deselect this checkbox to hide the advanced properties.

Default status: Deselected
Thread Pool Size Integer/Expression

Appears when you select Show Advanced Properties checkbox.

Specify the maximum number of threads to use to download multiple S3 objects simultaneously.

Default value: 10

Example: 5
Maximum Retries Integer/Expression

Appears when you select Show Advanced Propertiescheckbox.

Required. Specify the maximum number of retry attempts to make in case of a temporary network failure.

Default value: 3

Example: 5
Write Empty File Checkbox Appears when you select Show Advanced Properties checkbox.

Select this checkbox to write an empty S3 object when the incoming binary document has empty data. If there is no incoming document at the input view of the Snap, no S3 object is written regardless of the value in this field.

Default status: Deselected
Multipart Size (MB) Integer/Expression Appears when you select Show Advanced Properties checkbox.

  • The minimum multipart size is 5 MB.

  • The maximum multipart size is 5 GB.

  • The maximum number of parts in an S3 object is 10,000, and the maximum size of an S3 object is 5 TB.

  • To upload S3 objects that are more than 100 GB, you must increase the Multipart Size to at least the possible S3 object size divided by 10,000.

  • The larger the Multipart Size is, the larger the memory size that the Snap uses. For example, if the Multipart Size is 10 MB and the Thread Pool Size is 10, the memory size the Snap uses is 100 MB.

Learn more about uploading multipart files to S3.

Default value: 10

Example: 10
Access Control List Use this fieldset to define the Access Control List (ACL) to the specified S3 object.
Note:

  • The account that you use for the Snap must have the required permissions to set the ACLs for the S3 object it is uploading to.

  • S3 Express One Zone does not support ACL.
Grantee String/Expression/ Suggestion

Appears when you select Show Advanced Properties checkbox.

Select a grantee from the suggested list or enter a valid email address or a canonical ID associated to an AWS account. A canonical AWS ID can be obtained in the Security Credentials page of the AWS console.

Note: A grantee can be an AWS account or one of the predefined Amazon S3 groups.
Warning:

You can use email addresses to specify a grantee in only the following regions:

  • US East (N. Virginia)

  • US West (N. California)

  • US West (Oregon

  • Asia Pacific (Singapore)

  • Asia Pacific (Sydney)

  • Asia Pacific (Tokyo)

  • EU (Ireland)

  • South America (São Paulo)

Learn more about Specifying Grantee.

Default value: N/A

Example:

  • "Everyone"

  • "Authenticated user"

  • a canonical AWS ID: 1700891f3927e316dc4c9e18c789b32131880f48d3e03ac110aaf695b212573e
Read Checkbox Appears when you select the Show Advanced Properties checkbox.

Select this checkbox to grant permission to read the file.

Default status: Deselected
View Permissions Checkbox Appears when you select Show Advanced Properties checkbox.

Select this checkbox to grant permission to read the ACL.

Default status: Deselected
Full Control Checkbox Appears when you select Show Advanced Properties checkbox.

Select this checkbox to grant full control to the file.

Default status: Deselected
Canned ACL String/Expression Appears when you select Show Advanced Properties checkbox.

The predefined ACL grants (from AWS) to use when writing a file to S3.

Note: S3 Express One Zone does not support ACL.

Choose a Canned ACL from the available options:

  • None

  • Private

  • PublicRead

  • PublicReadWrite

  • AuthenticatedRead

  • LogDeliveryWrite

  • BucketOwnerRead

  • BucketOwnerFullControl

  • AwsExecRead

Watch the video for more information about AWS Canned ACL.Learn more about AWS Canned ACLs.

Default value: None.

Example: PublicRead
User-defined Object Metadata Use this field set to define key-value pairs for user-defined object metadata of an S3 object. Learn more about Using Metadata.
Note: This field set is not supported in S3 Express One Zone.
Key String/Expression Appears when you select Show Advanced Properties checkbox.

Specify the key name of the object metadata.

Note:

The key names of the object metadata are case-insensitive. AWS S3 converts them to lowercase and prefixes them with x-amz-meta- when displayed in the AWS S3 web console.

When the S3 File Reader Snap reads an S3 file, this metadata is shown in the header of the output binary data, and the key names are displayed in lowercase without the prefix “x-amz-meta-”.

Default value: N/A
Value String/Expression Appears when you select Show Advanced Properties checkbox.

Specify the value for the key entered above.

Default value: N/A
Object Tags Use this field set to define key-value pairs for object tags of an S3 object. Object tags enable you to categorize existing and new objects using key-value combinations. Learn more about Object Tagging.
Note: This field set is not supported in S3 Express One Zone.
Key String/Expression Appears on selecting Show Advanced Properties checkbox
Note: The key names of object tags are case-sensitive. When the S3 Download Snap reads an S3 object, these object tags are displayed in the header of the output binary data. If a key name of an object tag is similar to another tag in the header, then it is prefixed with “tag_”.

Default value: N/A
Value String/Expression Appears on selecting Show Advanced Properties checkbox.

Specify the value for the key entered above.

Default value: N/A
Pass Through Checkbox Appears on selecting Show Advanced Properties checkbox.

Select this checkbox to pass the input header as a value of the ‘original’ key in the output document.

Default status: Deselected
Snap execution Dropdown list
Choose one of the three modes in which the Snap executes. Available options are:
  • Validate & Execute: Performs limited execution of the Snap and generates a data preview during pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during pipeline runtime.
  • Execute only: Performs full execution of the Snap during pipeline execution without generating preview data.
  • Disabled: Disables the Snap and all Snaps that are downstream from it.

Default value: Execute only

Example: Validate & Execute

Examples