S3 File Writer
Overview
The S3 File Writer Snap is a Write-type Snap that reads a binary data stream from its input view and writes it to an S3 file destination. If you provide values for File permissions, the Snap sets the permissions to the file.
- This Snap has the ability to use an MD5 checksum that automatically checks for data integrity and corruption while uploading the file. If the checksum that Amazon S3 calculates during the upload does not match the value that Snap has sent in the request, S3 fails to store the object and the Snap displays an error.
- The current Snap functionality supports AWS S3 Cloud Service and is applicable for AWSGovCloud setup.
We plan to introduce additional S3 features exclusively in Amazon S3 Snaps, while Binary Snaps with S3 support will not contain these updates. Therefore, we recommend you to use the Amazon S3 Snap Pack for all your S3 operations within your pipelines. However, Binary Snaps will be retained as is to maintain backward compatibility, but be aware that we will no longer provide S3 support for the Binary Snaps.
Learn more: Migrate from Binary to S3 Snaps.

Read-type Snap
Works in Ultra Tasks
Snap views
| View | Description | Examples of upstream and downstream Snaps |
|---|---|---|
| Input | Any binary data stream. | |
| Output | If an output view is open and the file write action is successful, the output
view provides a document with information on the filename, result, and original
data. An example is: |
|
| Error |
Error handling is a generic way to handle errors without losing data or failing the Snap execution. You can handle the errors that the Snap might encounter when running the pipeline by choosing one of the following options from the When errors occur list under the Views tab. The available options are:
Learn more about Error handling in Pipelines. |
|
Snap settings
- Expression icon (
): Allows using pipeline parameters to set field values dynamically (if enabled). SnapLogic Expressions are not supported. If disabled, you can provide a static value.
- SnapGPT (
): Generates SnapLogic Expressions based on natural language using SnapGPT. Learn more.
- Suggestion icon (
): Populates a list of values dynamically based on your Snap configuration. You can select only one attribute at a time using the icon. Type into the field if it supports a comma-separated list of values.
- Upload
: Uploads files. Learn more.
| Field / Field set | Type | Description |
|---|---|---|
| Label | String | Required. Specify a unique name for the Snap. Modify this to be more appropriate, especially if more than one of the same Snaps is in the pipeline. Default value: S3 File Writer Example: S3 File Writer |
| File name | String/Expression/ Suggestion | Required. Specify the URL for the S3 file, from where
the binary data is to be read. This Snap also supports S3 Virtual Private Cloud
(VPC) endpoint. For example,
s3://my-bucket@bucket.vpce-028b7814794578709-vu0vvauy.s3.us-west-2.vpce.amazonaws.comNote:
The file name must start with s3:///. You can use the suggest
feature to view the list of buckets, subdirectories and files. Bucket names are
suggested if the property is empty or s3:///. Once you select a
bucket, it can list subdirectories and files immediately below the bucket. Names
of subdirectories end with a forward slash ("/"). The suggest feature is not
supported if the properties in the S3 Dynamic account are
parameters.Prerequisite: The provided account must have Read access to the specified S3 bucket in order to read the file successfully. Using Expressions: This property can be an
expression with the Expression enabler For example, if the File property is "s3:///mybucket/out_" + Date.now() + ".csv" then the evaluated filename is s3:///mybucket/out_2013-11-13T00:22:31.880Z.csv. Syntax: s3:///<S3_bucket_name>@s3.<region_name>.amazonaws.com/<path>For region names and their details, see AWS Regions and Endpoints. Warning: Region Name
Region name is optional only if the region is us-east-1. In all other cases the region name must be specified based on the syntax above. For example, mybucket@eu-west-1. For more information about regions, see AWS Regions and Endpoints. Refer to : Acceptable File Paths Warning: Lint Warning The Snap displays a
Lint Warning in your Pipeline in the following scenarios:
Therefore, we recommend that you confirm to any of the acceptable
relative paths. Otherwise, use an absolute path—that is provide a file path that
belongs to the same org where you want to write the file, or use the File Upload
Default value: s3:/// Examples:
|
| Suggest fully-qualified file names | Checkbox | Select this checkbox to include the region and authority of the S3 bucket in
the associated paths that appear in the Suggestion list. Note: We recommend you to
use fully-qualified suggestions from the Suggestions list if you are using an
instance in the gov cloud. Default status: Deselected |
| File action | Dropdown list | Required. Specify the action to perform if the file
already exists. The available options are:
Warning: Even though it is listed, Append is not supported for the
S3 file protocol. Default value: OVERWRITE Example: IGNORE |
| Write empty file | Checkbox | Select this checkbox to write an empty file when the incoming binary document
has empty data. If there is no incoming document at the input view of the Snap, no
file is written regardless of the value of the property. Default status: Deselected |
| Write header file | Checkbox | Select this checkbox to write a header file whose name is generated by
appending ".header" to the value of the File name property. The header file provides
metadata about the file such as content disposition or content type. The same header
information is also included in the output view data, as shown in the "Expected
output" section above, under the key "original". The binary data stream in the input view may contain header information about the binary data in the form of a document with key-value pair data. Tip: If the header has no keys
other than Content-Type or Content-Encoding, the .header file is not written.
Default status: Deselected |
| Validate after write | Checkbox |
Select this checkbox to allow the Snap to check if the file exists after writing the file. Default status: Deselected |
| Number of retries | Integer/Expression |
Specify the maximum number of retry attempts the Snap must make when it fails to write. If the value is larger than 0, the Snap first stores the input data in a temporary local file before writing to the target file. Tip:
Ensure that the local drive has sufficient free disk space as large as the expected target file size. If the value is larger than 0, the Snap first downloads the target file into a temporary local file. If any error occurs during the download, the Snap waits for the time specified in the Retry interval and attempts to download the file again from the beginning. When the download is successful, the Snap streams the data from the temporary file to the downstream Pipeline. All temporary local files are deleted when they are no longer needed. Minimum value: 0 Default value: 0 Example: 3 |
| Retry interval (seconds) | Integer/Expression | Specify the minimum number of seconds for which the Snap must wait before
attempting recovery from a network failure. Minimum value: 1 Default value: 1 Example: 3 |
| Buffer size (MB) | Integer |
Specify the data (in MB) to load into the S3 bucket, at a time. Warning:
Refer to Upload Part for more information on uploading to S3. Default value: 10 MB |
| Maximum upload threads | Integer | Specify the maximum number of threads to be used for the concurrent multipart
upload. The minimum value allowed is 1. Warning: To upload S3 files that
are more than 100 GB in size, we recommend you to set the Maximum upload
threads to 10. Also, set the Buffer size to 100 MB or more;
otherwise, many parallel HTTP connections may be opened, leading to the following
error: "AmazonClientException: Unable to execute HTTP request: Timeout waiting for
connection from pool". Default value: 10 Example: 3 |
| AWS Canned ACL | Dropdown list | The predefined ACL grants (from AWS) to use when writing a file to S3. Choose a
Canned ACL from the available options:
Watch the video for more information about AWS Canned ACL. Learn More: AWS Canned ACLs. Default value: None. Example: PublicRead |
| Access Control List | Use this field set to define the Access Control
List (ACL) to the specified S3 file. This filed set contains the following fields:
Warning: The account that you use for the Snap must have the
required permissions to set the ACLs for the S3 object it is writing to.
|
|
| Grantee | String/Expression/ Suggestion | Select a grantee from the suggested list or enter a valid email address or a
canonical ID associated to an AWS account. Canonical AWS ID can be obtained in the
Security Credentials page of the AWS console. Tip:
A grantee can be an AWS account or one of the predefined Amazon S3 groups. The following note is an excerpt from AWS document: "An email grantee is a grantee identified by their email address and authenticated by an Amazon system. email grants are internally converted to the canonical user representation when creating the ACL. If the grantee changes their email address, it will not affect existing Amazon S3 permissions. Adding a grantee by email address only works if exactly one Amazon account corresponds to the specified email address. If multiple Amazon accounts are associated with the email address, an AmbiguousGrantByEmail error message is returned. This happens rarely, but usually occurs if a user created an Amazon account in the past, forgotten the password, and created another Amazon account using the same email address. If this occurs, the user should contact Amazon customer service to have the accounts merged. Alternatively, grant user access specifying the canonical user representation." Warning: "Everyone” option allows anyone in the world to access
the file (authenticated or anonymous). Even when the bucket is protected with
permission and if the file operation in the Snap under ACL->Grantee is set to
'Everyone', any user (authenticated or anonymous) can access the file. So, we
highly recommend not to use this option as its unsafe. Warning: Email Address Grantee Access You can use Email addresses to specify a
grantee only in the following regions:
For more information, see Specifying Grantee. Examples:
|
| Read | Checkbox | Grants permission to read the file. |
| View permissions | Checkbox | Grants permission to read the ACL. |
| Full control | Checkbox | Grants full control to the file. |
| User-defined object metadata | Use this field set to define key-value pairs for
user-defined object metadata of an S3 object. For more information about
user-defined object metadata, see Using Metadata This field set contains the
following fields:
|
|
| Key | String/Expression | Specify the key name of the object metadata. Tip:
The key names of the object metadata are case-insensitive. AWS S3 converts them to lower-case and prefixes them with “x-amz-meta-” when displayed in the AWS S3 web console. When the S3 File Reader Snap reads an S3 file, this metadata is shown in the header of the output binary data, and the key names are displayed in lower-case without the prefix “x-amz-meta-”. |
| Value | String/Expression | Specify the value for the key entered above. |
| Object tags | Use this field set to define key-value pairs for
object tags of an S3 object. Object tags enable you to categorize existing and new
objects using key-value combinations. For details about the object tags, see Object Tagging. This field set contains the
following fields:
|
|
| Key | String/Expression |
Specify the key name of the object tag. Tip:
The key names of object tags are case-sensitive. When the S3 File Reader Snap reads an S3 file, these object tags are displayed in the header of the output binary data. If a key name of an object tag is the same as another in the header, it is prefixed with “tag_”. See example Providing User-defined Object Metadata and Object Tags using the S3 File Writer Snap below for more information. |
| Value | String/Expression | Specify the value for the key entered above. |
| Snap execution | Dropdown list |
Choose one of the three modes in
which the Snap executes. Available options are:
Default value: Execute only Example: Validate & Execute |
Acceptable File Paths
- Relative paths
filename.json: Saves the file in the project.../shared/filename.json: Saves the file in the Project Shared Space.../../shared/filename.json: Saves the files in the Org Shared project.
- Absolute path
-
/<org>/<projectSpace>/<project>/filename.json
-
Optional Configuration
To access S3 files from Groundplex nodes hosted in the EC2 environment without specifying Access-key ID and Secret key in AWS S3 account configured for the Snap, enable the ‘IAM_CREDENTIAL_FOR_S3’ feature. When you enable this feature, the IAM credential stored in EC2 metadata is used to access S3 buckets.
- Open Manager
- Open the Snaplexes tab of the project that contains the EC2-based Groundplex.
- Click the Groundplex to open its Properties.
- Open the Node Properties tab.
- Add a new row in the Global properties section.
- Specify
jcc.jvm_optionsas the Key and-DIAM_CREDENTIAL_FOR_S3=TRUEas the Value.
Restart the JCC (node) to apply the changes. For more information about IAM Roles, refer to IAM Roles for Amazon EC2.