Overview

You can use this Snap to read data from various sources (such as SLDB, HTTP, S3, SFTP, HDFS, etc.) and produce a binary data stream at the output.

• Read-type Snap
• Works in Ultra Tasks

Prerequisites

IAM Roles for Amazon EC2

The 'IAM_CREDENTIAL_FOR_S3' feature is used to access S3 files from EC2 Groundplex, without Access-key ID and Secret key in the AWS S3 account in the Snap. The IAM credential stored in the EC2 metadata is used to gain access rights to the S3 buckets. To enable this feature, set the Global properties (Key-Value parameters) and restart the JCC:jcc.jvm_options = -DIAM_CREDENTIAL_FOR_S3=TRUE

This feature is supported in the EC2-type Groundplex only. Learn more.

Connect to FTP server:

To connect to the FTP server that needs to reuse the session for data transfer over TLS protocol, add:

-DFTPS_SSL_TLS_PROTOCOL=TLSV1.2  (or) TLSV1.3

property as a JVM option under the Global properties of the Node Properties tab:

Limitations

Known issues

Snap views

[ { "": "Preview binary0...", "content-type": "text/html; charset=utf-8", "x-frame-options": "DENY",
 "connection": "keep-alive", "transfer-encoding": "chunked", "date": "Thu, 23 Oct 2014 00:24:40
GMT", "content-location": "https://www.facebook.com", "pragma": "no-cache", "p3p": "CP=\"Facebook
does not have a P3P policy. Learn why here: http://fb.me/p3p\"", "cache-control": "private, no-
cache, no-store, must-revalidate", "x-xss-protection": "0", "x-content-type-options": "nosniff", "x-
fb-debug":
"N6wiHWAvz9kzpPUoM5vTm+yZzCZyiSrHXFXumHQixfMd0Qi+VDm514PkrrmQu2ISuuMTTFtUTqDZgDVG4blPTw==",
"expires": "Sat, 01 Jan 2000 00:00:00 GMT", "set-cookie": "reg_ext_ref=deleted; expires=Thu, 01-Jan-
1970 00:00:01 GMT; Max-Age=0; path=/; domain=.facebook.com" } ]

Snap settings

Field / Field set	Type	Description
Label	String	Required. Specify a unique name for the Snap. Modify this to be more appropriate, especially if more than one of the same Snaps is in the pipeline. Default value: File Reader Example: File Reader
File	String/Expression	Required. Specify the URL for a regular file that must begin with a file protocol. The supported file protocols are: http: https: s3: ftp: ftps: sftp: hdfs: sldb: smb: file: (only for use with a Groundplex) wasb: wasbs: gs: adl: You can also upload a file from using the Upload icon. You can preview the uploaded file using the preview icon. Learn more about Previewing File Note: This Snap supports S3 Virtual Private Cloud (VPC) endpoints. For example: s3://my-bucket@bucket.vpce-028b7814794578709-vu0vvauy.s3.us-west-2.vpce.amazonaws.com This Snap supports Oracle Object Storage endpoints when used with pre-authenticated requests. For example: https://objectstorage.us-sanjose-1.oraclecloud.com/p/123AbcdEFG12345_xyz123/n/MyNamespace/b/snaplogic-academy/o/sample.json To create a pre-authenticated request, refer to the instructions in the following Oracle article: Using Pre-Authenticated Requests. Learn more about : Reading files from Project and Shared Project Spaces File value as an Expression Warning: The File value should be an absolute path for all protocols except for SLDB. For files in SLDB, the Snap can read only files in the same Project Directory or the Shared Project Directory. It cannot access files from other Projects. Typically, the file names in the Reader Snaps are read from incoming document which might have a structure different from the relative path. For optimal results, we recommend that you build absolute paths to their projects and then add the file name. Note: When you provide a file path that contains more than five entities (for example, entity1/entity2/entity3/entity4/file1.json) the Snap displays a Lint Warning in your Pipeline. Note: "://" is a separator between the file protocol and the rest of the URL and the host name and the port number should be between "://" and "/". The hostname and port number are omitted in the SLDB and s3 protocols. If the port number is omitted, a default port for the protocol is used. The file:/// protocol is supported only on Groundplex. In Cloudplex configurations, use SLDB or other file protocols. When using the file:/// protocol, the file access is done using the permissions of the user assigned or associated with the Snaplex (by default Snapuser). File system access is to be used with caution, and it is the customer's own responsibility to ensure that file system is cleaned up after use. Default value: N/A Example: "asset.json" or "sldb:///asset.json" "shared/asset.json" or "sldb:///shared/asset.json" s3://my-bucket@bucket.vpce-028b7814794578709-vu0vvauy.s3.us-west-2.vpce.amazonaws.com s3:///<S3_bucket_name>@s3. <region_name>.amazonaws.com/<path>/<file_name> For region names and their details, see AWS Regions and Endpoints. Example: s3:///[email protected]/test.json sftp://ftp.snaplogic.com:22/dir/filename smb://smb.snaplogic.com:445/test_files/csv/input.csv $filename (The value of the $filename is obtained from the input document and the document should have an entry with the "filename" key.) _filename (A key/value pair with "filename" key should be defined as a pipeline parameter.) file:///D:/testFolder/ (if the Snap is executed in the Windows Groundplex and needs to access D: drive) wasb:///Snaplogic/testDir/sample.csv (to read 'sample.csv' file in the 'testDir' folder in the 'Snaplogic' container) gs:///mybucket/csv/test.csv (to read 'test.csv' file in the 'csv/' folder of the 'mybucket' bucket) adl://storename/folder/filename (to read the file from a location of the storage)
Prevent URL encoding	Checkbox	When enabled, this will prevent the Snap from automatically URL encoding the file path (including the query string if it exists). Enable this setting to use the file path value as-is. Refer to section : Encoding of Characters in a URL Default status: Deselected
Enable staging	Checkbox	If selected, the Snap downloads the source file into a local temporary file. When the download is completed, it streams the data from the temporary file to the output view. This property prevents the Snap from being blocked by slow downstream pipeline. The local disk should have sufficient free space as large as the expected file size. Warning: Some Snaps may take a long time to process large amounts of data. This, in turn, could lead to connection timeouts, causing the pipeline to fail. Selecting this property saves the data on your local disk, enabling you to avoid such timeouts. Default status: Deselected
Number of retries	Integer/Expression	Specify the maximum number of retry attempts that the Snap must make in case there is a network failure, and the Snap is unable to read the target file. Note: If the value is larger than 0, the Snap first downloads the target file into a temporary local file. If any error occurs during the download, the Snap waits for the time specified in the Retry interval and attempts to download the file again from the beginning. When the download is successful, the Snap streams the data from the temporary file to the downstream Pipeline. All temporary local files are deleted when they are no longer needed. Ensure that the local drive has sufficient free disk space to store the temporary local file. Minimum value: 0 Default value: 0 Example: 3
Retry interval (seconds)	Integer/Expression	Specify the minimum number of seconds for which the Snap must wait before attempting recovery from a network failure. Minimum value: 1 Default value: 1 Example: 3
Additional output headers	Use this field set to define key-value pairs as additional headers in the binary output. Note: If the downstream Snap is HTTP Client Snap, these headers are used to evaluate expressions and perform the multipart file upload from various file protocols multiple times.
Key	String	Specify the key name for the additional output header.
Value	Dropdown list/Expression	Specify the value for the additional output header.
Advanced properties	Use this field set to define specific settings for polling files. Click to add a new row for defining an advanced property. This field set contains the following fields: Properties Values
Properties	Dropdown list	The URI of the Shared Access Storage (SAS) to be accessed. Supported SAS types are: Service SAS on container Service SAS on blob Account SAS
Values	String/Expression	Specify the value for the SAS URI. Warning: Ensure that the URI is specified in the format described here. If the SAS URI value is provided in the Snap settings, then the settings provided in the account (if any account is attached) are ignored. Default value: N/A Example: https://myaccount.blob.core.windows.net/sascontainer/sasblob.txt?sv=2015-04-05&st=2015-04-29T22%3A18%3A26Z&se=2015-04-30T02%3A23%3A26Z&sr=b&sp=rw&sip=168.1.5.60-168.1.5.70&spr=https&sig=Z%2FRHIX5Xcg0Mq2rqI3OlWTjEg2tYkboXr1P9ZUXDtkk%3D
Snap execution	Dropdown list	Choose one of the three modes in which the Snap executes. Available options are: Validate & Execute: Performs limited execution of the Snap and generates a data preview during pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during pipeline runtime. Execute only: Performs full execution of the Snap during pipeline execution without generating preview data. Disabled: Disables the Snap and all Snaps that are downstream from it. Default value: Execute only Example: Validate & Execute

Reading files from Project and Shared Project Spaces

• If a Pipeline is created in a project other than the shared project and you want to read the "asset.json" file from the same project, enter "asset.json" or "sldb:///asset.json".
• If a Pipeline is created in the shared project and you want to read the "asset.json" file from the shared project, enter "asset.json" or "sldb:///asset.json".
• If a Pipeline is created in a project other than the shared project and you want to read the "asset.json" file from the shared project, enter "shared/asset.json" or "sldb:///shared/asset.json".
• Ensure the file name, folder name, or the file path does not contain '?' character because it is not fully supported and when present, the Snap might fail.

File value as an Expression

The File value can be a JavaScript expression which is evaluated with values from the input view document and the pipeline parameters. The syntax for file value is: [protocol]://[host][:port]/[path]

• $filename (The value of the $filename is obtained from the input document and the document should have an entry with the "filename" key.)
• _filename (A key/value pair with "filename" key should be defined as a pipeline parameter.)

Encoding of Characters in a URL

Following are some of the common characters that are automatically encoded by the Snap:

Following are some of the characters that are not automatically encoded by the Snap:

Preview File

To preview a file, in the File click the Preview icon.

Troubleshooting

Examples

• HDFS
• SFTP File Read