REST Get [Not Recommended]

Overview

You can use the REST Get Snap to execute an HTTP Get method on a REST API service endpoint.
Note: We highly recommend that you use the HTTP Client Snap instead of REST Snaps to take advantage of the Capabilities of HTTP Client Snap. Learn more about Migrate from REST Snaps to HTTP Client Snap.

Rest Get Overview

Limitations

The REST GET Snap Link HTTP response header maybe be malformed compared to other third-party applications like POSTMAN.

Snap views

View Description Examples of upstream and downstream Snaps
Input Each input document offers details associated with the data required from the target RESTful server.
Output Each output document contains data retrieved from the RESTful web server.

If the Snap fails during the operation, it sends an error document to the error view containing the fields error, reason, resolution, and stacktrace. However, you must enable the Error view to view the error document.

Error

Error handling is a generic way to handle errors without losing data or failing the Snap execution. You can handle the errors that the Snap might encounter when running the pipeline by choosing one of the following options from the When errors occur list under the Views tab. The available options are:

  • Stop Pipeline Execution Stops the current pipeline execution when an error occurs.
  • Discard Error Data and Continue Ignores the error, discards that record, and continues with the remaining records.
  • Route Error Data to Error View Routes the error data to an error view without stopping the Snap execution.

Learn more about Error handling in Pipelines.

Snap settings

Legend:
  • Expression icon (): Allows using JavaScript syntax to access SnapLogic Expressions to set field values dynamically (if enabled). If disabled, you can provide a static value. Learn more.
  • SnapGPT (): Generates SnapLogic Expressions based on natural language using SnapGPT. Learn more.
  • Suggestion icon (): Populates a list of values dynamically based on your Snap configuration. You can select only one attribute at a time using the icon. Type into the field if it supports a comma-separated list of values.
  • Upload : Uploads files. Learn more.
Learn more about the icons in the Snap settings dialog.
Field / Field set Type Description
Label String Specify a name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your pipeline.

Default value: REST Get

Example: Get User Details
Service URL String/Expression

Required. Specify the service endpoint URL of REST API. You can provide the URL in one of the following ways:

  • As a JavaScript expression in expression language.

    http://coresnapsqa-v4.s3.us-east-2.amazonaws.com/S3Account/'+encodeURIComponent("Special_char_owner_!3)12.json")

  • As a plain string that you should enclose in double quotes ("").

    "https://www.example.com/slm/webservice/v2.0/defect/%s".replace("%s", $.widget.id)

For Snaps using AWS Signature V4 accounts, you can use the canonical name (CNAME) for the URI so it's not necessary for the URL to end with amazonaws.com or have the region and service provided in it. However, if you are using the CNAME you must provide it in the AWS Region and Service Name fields in the AWS Signature V4 account.

The host name in the CNAME must be equal to the bucket name.

For example:

  • Previous URI: https://jsmith-bucket.nia3.snaplogic.net.s3.us-west-1.amazonaws.com/HttpClientTest/test1
  • New URI with CNAME: https://jsmith-sltest.nia3.snaplogic.net/HttpClientTest/test1

Here, the bucket name is jsmith-bucket.nia3.snaplogic.net

The Snap finds the value at the JSON path $.widget.id in the input data and replaces "%s" in the Service URL with the value. You can connect File Reader and JSON Parser Snaps upstream of a REST Put Snap and prepare the following JSON file for the File Reader Snap:

[ 
{ "widget": { "id": "111", "name": "foo" } } 
{ "widget": { "id": "555", "name": "bar" } } ] 
"https://www.example.com/data/v27.0/sobjects/Account"
Note:

The Service URL for a REST snap must be valid. If the Service URL contains any special characters, such as !, =, %21, $, and ^, the Snap throws an exception error. You can escape the special characters (using expression language) with one of the following methods:

  • Use the global function encodeURIComponent on any variables that might contain special characters so that they are encoded properly. For example:
    http://coresnapsqa-v4.s3.us-east-2.amazonaws.com/S3Account/'+encodeURIComponent(_fileName)
  • If you are aware of the code, use the code directly in the Service URL field. For example (where <%213%2912> is the code):

    http://coresnapsqa-v4.s3.us-east-2.amazonaws.com/S3Account/Special_char_owner_%213%2912.json

We recommend that you use the encodeURIComponent method to escape the special characters.

Default value: N/A

Example: https://elastic.snaplogic.com/api/1/rest/public/runtime/snaplogic?start=1430377200000&end=1430420399000
Process array Checkbox
Select to produce a stream of documents with one record in each document.
  • If the Process Array property is selected, it takes a response that’s an array [{"value": 1}, {"value": 2}] and provides the individual inner json as separate documents. If the content provided doesn’t parse with the provided Content-Type, the snap will error.
  • If the Process Array property is not selected, the Snaps attempts to convert the entity to a string or a byte array depending on the response headers.

Default value: Deselected

Trust all certificates Checkbox

Select this checkbox to trust all certificates, such as self-signed certificates.

Default value: Deselected

Follow redirects Checkbox

Select this checkbox to enable the Snap to follow redirects.

  • If you select this checkbox, the Snap receives a redirect response, such as a 301 or 302 response code, it sends another request to the URL specified in the response Location header.
  • If you deselect this checkbox, the Snap does not make another request, and the 301/302 response appears in the output view.

Default value: Selected

Query parameters

Use this field set to add query parameters to your request. This field set comprises the following fields:

  • Query parameter
  • Query parameter value
Query parameter String The name of the query parameter.

Default value: N/A

Example: id
Query parameter value String/Expression The value that you want to assign to the parameter.
Note: This property is expression-enabled. For more information on the expression language, see Understanding Expressions in SnapLogic and Using Expressions. For information on pipeline parameters, see Pipeline Properties.

Default value: N/A

Example: $widget.id
HTTP Header

Use this field set to create the HTTP header key-value pairs required for defining the headers for different types of input (JSON, PDF, DOCX, and so on). If you want to specify only content-type headers, you can configure the Multipart Content-Type property instead. This field set comprises the following fields:

  • Key
  • Value
Key String/Expression The name of the HTTP header.
Note: This property is expression-enabled. For more information on the expression language, see Understanding Expressions in SnapLogic and Using Expressions. For information on pipeline parameters, see Pipeline Properties.

Default value: N/A

Example: content-type
Value String/Expression The value that you want to assign to the HTTP header.
Note: This property is expression-enabled. For more information on the expression language, see Understanding Expressions in SnapLogic and Using Expressions. For information on pipeline parameters, see Pipeline Properties.

Default value: N/A

Example: application/json
Response entity type Dropdown list Select one of the following response entity types you want the Snap to display in the output document.:
  • DEFAULT - Select this option to enable the Snap to process the response automatically. The response entity is processed automatically based on the Content-Type header in the response. If the content provided does not parse with the provided Content-Type, the snap will result in an error.
    The following two scenarios describe the Snap’s behavior when using the Process Array checkbox:
    • When you select the Process Array checkbox, the Snap parses JSON or XML and produces a stream of documents at the output view.
    • When you deselect the Process Array checkbox, the Snaps sends the data as it is at the output view.
  • TEXT - Select this option to enable, the Snap to produce an entity of string type.
  • BINARY - Select this option to enable the Snap to produce an entity of byte array type.
Note: If you select TEXT or BINARY, the Snap does not parse the entity content. If you select DEFAULT, the Snap produces the expected result in most cases, but if it fails to process as expected, you can set the Response entity type to TEXT or BINARY.

Default value: DEFAULT

Example: TEXT
Cookie Policy String/Expression/ Suggestion Select a Cookie Policy from the following options:
  • Browser Compatibility: This policy is compatible with different servers even if they are not completely standards-compliant. If you are facing issues while parsing cookies, you should try using this policy.
  • Ignore Cookies: This cookie policy ignores all cookies. You should use this policy to prevent HTTP Client from accepting and sending cookies.
  • RFC Strict: This policy uses the set-cookie header
  • RFC Lax: The policy uses set-cookie and set-cookie2 for parsing.

Default value: Ignore Cookies

Example: RFC Strict
Read timeout Integer Specify the number of seconds for which the Snap waits before aborting the request due to a failure to read from the target service.
  • In some cases, the request does not return, such as during network issues. For those a timeout can be configured to allow failing fast.
  • Every Snap instance can have its own timeout settings.
  • The Snap will execute five retries within a couple of seconds back-off time before it fails.

Default value: 900

Example: 60
Connection timeout Integer Specify the number of seconds for which the Snap waits before aborting the request due to a failure to establish a connection to the target endpoint or service.

Default value: 30

Example: 60
Maximum request attempts Integer Specify the maximum number of attempts that the Snap must make to receive a response. If the attempts do not result in a response, the Snap terminates the request.

Default value: 5

Example: 3
Retry request interval Integer Specify the time in seconds to wait before retrying the request.

Default value: 3

Example: 10
Retry Policy String Select how you want to handle connection and error responses from the following options:
  • Connection errors: Occurs when a client fails to establish a secure connection to the server.
  • Connection and auth errors (401/403): Occurs when either the user authorization fails or access to the page/resource is forbidden.
  • Connection and client errors (4xx): Occurs when the request for the resource contains bad syntax or when the resource is not found (404).
  • Connection and server errors (5xx): Occurs when the server is unable to complete a request.
  • All errors: Applies to all of the above-listed errors.

Default value: Connection errors

Example: All errors
Normalize URL Checkbox Select this checkbox to normalize the Service URL. This enables the Snap to convert double slashes (//) in the URL path to single slash (/).

For example, https://example.com/path//to//file is converted to https://example.com/path/to/file.

Deselecting this checkbox reverts the Snap to 4.19 Snaplex behavior, wherein the URL paths were not normalized by default.

Note: In the 4.20 release, due to the HTTP client upgrade the URL paths were normalized by default. Hence, there was a change in behavior in handling the URL paths in 4.20 release when compared to 4.19. This change in behavior should not impact the existing pipelines, because most of the websites map URL paths with double or single slashes to the same endpoint. For example, https://snaplogic.com/company/diversity and direct to the same endpoint. Therefore, we recommend you to select the Normalize URL checkbox.

However, an exception to this is when you use non-standard URLs that differentiate the URL paths containing double slashes from those with single slashes and map them to different endpoints, in which case you must deselect Normalize URL checkbox.

For example, http://host/pages/foo.html and http://host/pages//foo.html point to different URIs, and servers assign different meanings to them.

Default value: Selected

Example: Deselected
Has next String/Expression Enter the expression that must be evaluated on the output document to true or false, to indicate whether there are more records available. The Snap continues to read the next page until this expression evaluates to false in the new set of output documents.
Note:

Has next is a property in the API response that indicates whether there are more records to be fetched. This property is used for the REST Get pagination feature, which retrieves 20 records at a time (referred to as a page), by default. Learn more about REST Get Pagination.

To see this property in action, see Retrieve Contact Information from HubSpot or Retrieving Contact Information from HubSpot Using Upstream Values, below.

Note: This field is null-safe; if the evaluated value is null, or if there is no value provided in Has next field, the Snap stops the pagination.

Default value: N/A

Example: $entity['has-more']
Next URL String/Expression Enter the expression that is evaluated on the output document to a URL that the Snap must use to get the next page. This property is used for the REST Get pagination feature.

The Next URL has two components, which are structured as follows:

"<endpoint_URL>&<offset_indicator>=" + $entity['last_ID_retrieved']

  • The first component contains the URL of the endpoint and an instruction to look for the offset indicator. ("<endpoint_URL>&<offset_indicator>=").
  • The second component is a pointer to the parameter in the API response that contains the ID of the last entity retrieved. This tells the endpoint that the next page must contain entities that are returned after the entity number listed in the Next URL.

You can create the Next URL in either of the following ways:

  • You can enter the entire URL in the expression and include the parameter required to fetch the next set of documents.

    Example: "https://api.hubapi.com/contacts/v1/lists/all/contacts/all?hapikey=demo&vidOffset=" + $entity['vid-offset']

  • If you have the URL part of the Next URL as an input from an upstream Snap, you can structure the Next URL expression using the original parameter in the API response received. For example, if you have the following input coming from upstream:
    [
     {
      "url": "https://api.hubapi.com/contacts/v1/lists/all/contacts/all?hapikey=demo&vidOffset"
     }
    ]
You can configure the Next URL field as follows:
$original.msg+ "=" + $entity['vid-offset']

Here, vid-offset is the API response parameter containing the ID of the last output received from the endpoint.

If the API response does not contain a field with information about the last entity retrieved, you can use the Snap's output document count to fetch the next page.

For example, https://domain.com/object/page/' + (snap.out.totalCount + 1).toString()

Note: The Next URL is evaluated as an expression but not encoded, which is the general behavior of the Snap.

Default value: N/A

Example: $entity['has-more']
Pagination interval(s) Integer The number of seconds for the Snap to wait before attempting to get the next page.

Default value: 0

Example: 10
Show all headers Checkbox The REST endpoint may return a response with multiple headers with the same header name.
  • If this property is deselected, only one response header will be shown in the output document.
  • If selected, the response header displays all the response header values received as a list of objects.
Note: If any of these objects has a key-value format, it is parsed into the map data.

Default value: Deselected

Snap execution Dropdown list Select one of the following three modes in which the Snap executes:
  • Validate & Execute: Performs limited execution of the Snap, and generates a data preview during Pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during Pipeline runtime.
  • Execute only: Performs full execution of the Snap during Pipeline execution without generating preview data.
  • Disabled: Disables the Snap and all Snaps that are downstream from it.

Default value: Validate & Execute

Example: Validate & Execute

REST Get Pagination

When making REST API calls to a specific endpoint using the GET method, there might be a lot of results to return. We use the pagination settings to fetch all the records/documents on multiple pages. For example, if the GET API call returns 20 documents by default, to get all other pages/records, we use Has Next field and get the list of all records until the last record.

Using REST Get to Process gzip Compressed Data

The response can provide gzipped data. In that case, the Snap can be configured to process it using the header parameter:

HTTP Header

Key Value
Accept-Encoding gzip
Accept */*

The stream is then uncompressed while parsing the data and making it available on the output view.

Troubleshooting

Failed to execute HTTP request

Reason: The Service URL must have a protocol, such as http:// or https://. <Service_URL>: Name or service not known. This basically means that the service URL you entered is not accessible.

Resolution: Check the Snap properties. Specifically, check your entry in the Service URL field.

Failure: Validation errors: property_map.settings.serviceUrl.value: Could not compile expression: <Service_URL>

Reason: The snap settings are not correct. Encountered extraneous input ':' at line 1:5; expecting={<EOF>, '||', '&&', '^', '==', '!=', '>', '<', '>=', '<=', '+', '-', '*', '/', '%', '?', 'instanceof', 'in', 'matches', '[', '(', '.', PropertyRef}.

This means that you have turned expressions on () while entering string value into the Service URL field.

Resolution:Fix the snap configuration and try executing again. Please check expression syntax. Ensure that your service URL is a proper expression; or turn off the expression control () .

URL Parse Exception - 403

Reason: The Service URL path might be containing any of the following special characters: !, =, %, #, $, ^&()_¢äâêîôûñç¡¿ÉÙËǨ°¸ðø©¢¾A+²½µ®§÷¶þ

Resolution: To escape the special characters, use the global function encodeURIComponent on any variables that might contain special characters so that they are encoded properly.

Too many Requests - 429

Reason: There are too many requests to REST endpoint.

Resolution:Wait for the retry to succeed. Rest related Snaps extract the response header and automatically retry when they encounter status 429.

By default, the retry interval (Retry-After) is specified in the HTTP response header. If no value is available for Retry-After, then the Snap’s Retry Interval value is used.

Example

  • Create and Delete an Account in SnapLogic Using REST Snaps
  • Retrieve Contact Information from HubSpot
  • Retrieving Contact Information from HubSpot Using Upstream Values
  • Connect to the SnapLogic API
  • Connect to rally with REST
  • REST Get pagination: Eloqua and Marketo
  • Handles Multiple Response Cookies
  • When Process Array is Selected or Deselected