S3 Account

Overview

You can use this account type to connect S3 Snaps with data sources that use an Amazon S3 account.



Prerequisites

  • Either EC2 or ECS instance as a Groundplex. The IAM role is valid only in Groundplex nodes hosted in the EC2 or ECS environment.

  • JCC with the following global property set:jcc.jvm_options=-DIAM_CREDENTIAL_FOR_S3=TRUE

    Note: If you do not have an EC2 instance Groundplex, you can authenticate your account using the Access Key ID and Secret Key. You can assume roles using the Cross account IAM role, which uses the IAM role specified in the settings. The Access Key ID and Secret Key need to be able to assume roles in the user specifications.

Limitations

  • MinIO does not support ACL. Canned ACL in S3 Upload Snap does not work.

  • Cross-account IAM Role is not supported.

  • Glacier storage classes are not supported in S3 Archive and S3 Restore Snaps.

Other S3-compatible versions supported by the AWS SDK may work with the S3 Snaps; however, customers must verify this.

Account settings

Legend:
  • Expression icon (): Allows using JavaScript syntax to access SnapLogic Expressions to set field values dynamically (if enabled). If disabled, you can provide a static value. Learn more.
  • SnapGPT (): Generates SnapLogic Expressions based on natural language using SnapGPT. Learn more.
  • Suggestion icon (): Populates a list of values dynamically based on your Snap configuration. You can select only one attribute at a time using the icon. Type into the field if it supports a comma-separated list of values.
  • Upload : Uploads files. Learn more.
Learn more about the icons in the Snap settings dialog.
Field / Field set Type Description
Label String

Required. Specify a unique label for the account.

Default value: None.

Example: S3 Account
Access Key ID String/Expression Specify a unique access key ID part of AWS authentication.
Note: This field is required if the IAM role is disabled.

Default value: N/A

Example: CKIA2EP4BT3EYCWBKC
Secret Key String/Expression

Specify the secret key part of AWS authentication.

Note: This field is required if the IAM role is disabled.

Default value: N/A

Example: G9Hm2h5+PtSI7CnZO9KLVgyFPAc5ZTqC9uV94uPr
Secret key String/Expression

Appears when User credentials is selected for Authentication type field.

Specify the secret key.

Default value: N/A

Example: Encrypted
Security token String/Expression

Specify the security token that is part of AWS Security Token Services (STS) credentials. Note that only global STS regions are supported.

Default value: N/A

Example: Djh198SnOKIA2
IAM Role Checkbox

Select this checkbox to use the EC2 or ECS instance stored in the IAM role, instead of the normal AWS authentication to access the S3 bucket. The Access-key ID and Secret key fields are ignored in this case. Learn more about Configuring an EC2 role for IAM Role in AWS S3 Account .

Important: The Amazon S3 Snap Pack automatically detect the Maximum session duration value for the Cross-Account IAM role (1 through 12 hours). The Snaps round down the value to the nearest hour. So, if the Snap administrator sets the Maximum session duration at 3 hours and 45 minutes, the Snaps read it as 3 hours. The Snaps also refresh the session before it expires. However, the automatic session refresh does not support the case of very large file upload or download that takes longer than the maximum session duration.
Note:
  • The IAM role is valid in Groundplex nodes hosted in the EC2 and ECS environments.
    • For EC2, set the global properties (Key and Value) on the Groundplex as shown below and restart the JCC: 
      jcc.jvm_options = -DIAM_CREDENTIAL_FOR_S3=TRUE

    • For ECS, update the task definition environment variables as follows:

      [

{
                    "name": "LANG",
                    "value": "en_US.UTF-8"
                },
                {
                    "name": "LC_ALL",
                    "value": "en_US.UTF-8"
                },
                {
                    "name": "JAVA_TOOL_OPTIONS",
                    "value": "-Dfile.encoding=UTF-8 -Dsun.jnu.encoding=UTF-8"
                }
            ]

  • The List, Read, and Write permissions are required as per the attached S3 policy for the IAM role stored on the EC2 and ECS instances.

  • When you select the IAM Role checkbox and validate the account, an error is displayed, and the account is not validated. To use the account, provide valid Role ARN and External ID values, and then click the Apply button.

Default status: Deselected
Cross Account IAM Role

Configure the properties required to perform cross-account access. Learn more about Configuring Cross Account IAM Role Support .

Note: When you use the Cross Account IAM Role in the Amazon S3 account, the ownerDisplayName and ownerID fields in the output document of the S3 Browser Snap may be empty.
Role ARN String/Expression

Specify the Amazon Resource Name of the role to assume.

Default value: N/A

Example: arn:aws:s3:::sandbox-test-snap
External ID String/Expression

Specify an optional external ID that might be required by the role to assume.

Default value: N/A

Example: 321f248c-8f4a-21be-87c4-184c9f8e2d03
Encryption

Configure the properties required for encryption.
Encryption type Dropdown list/Expression Specify the AWS Key Management Service key used to encrypt S3 objects. It can be the key ID or ARN. The available options are:

  • None: The files do not get encrypted using KMS encryption.

  • Server-Side Encryption: The S3 file is written and encrypted using the 256-bit Advanced Encryption Standard AAES256. For Snaps that read objects from S3, this field is not required, because encrypted data is automatically decrypted when data is read from S3.

  • Server-Side KMS Encryption: The output files on Amazon S3 are encrypted using this encryption with an Amazon S3-generated KMS key.

  • Client-Side KMS Encryption: The output files on Amazon S3 are encrypted using this encryption with a client-generated KMS key.

Note:
  • For Snaps that read objects from S3, this field is not required.
  • For Snaps that write objects to S3, this is required for encryption types—Server-Side encryption and Client-Side encryption with AWS KMS-Managed Keys.

    • For Server-Side encryption, the key must be in the same region as the S3 bucket.

    • For Client-Side encryption, a key from any region can be used by using the key ARN value. If a key ID is used for Client-Side encryption, it defaults to the us-east-1 region.

Default value: None.

Example: Server-Side Encryption
KMS Key String/Expression

Specify the AWS Key Management Service (KMS) key ID or ARN to be used for the S3 encryption.

Note:

  • This is required only if the KMS Encryption type property is configured to use the encryption with KMS. Learn more about AWS KMS Overview and Using Server Side Encryption.

  • For Snaps that write objects to S3, this is required for encryption types Server-Side encryption with AWS KMS-Managed Keys and Client-Side encryption with AWS KMS-Managed Keys.

    • For Server-Side encryption, the key must be in the same region as the S3 bucket.

    • For Client-Side encryption, a key from any region can be used by using the key ARN value. If a key ID is used for Client-Side encryption, it defaults to the us-east-1 region.

Default value: None.

Example: 28e3c2b6-74e2-4a3e-9890-6cd8e1c03661
KMS Region String/Expression/ Suggestion Specify the AWS region where the KMS key is located.

Default value: N/A

Example: s3.us-east-2
S3 endpoint String/Expression Specify the S3 endpoint URL to connect to an S3-compatible server. You can also configure the S3 endpoint in Global properties of Node Properties as follows:

jcc.jvm_options = -Ds3.endpoint=http://192.168.0.209:9000/

Note: When you define values in both (global properties and an S3 account), the account settings override global settings.

Default value: N/A

Example: http://192.168.0.209:9000
Bucket for account validation String Specify the bucket for validating the account.

Troubleshooting

Error Message

Error Reason Resolution
When authorizing an S3 account, if the IAM role checkbox is selected, the following error is displayed on clicking the Validate button.

"Failed to validate account."

 When validating the S3 account if the IAM role checkbox is selected, an error is displayed. Account validation is not supported when you select the IAM Role checkbox.Ensure that you provide valid Role ARN and External ID values and then click Apply instead of Validate (on the account settings dialog) to authorize and use the account.