Pipeline requirements for Ultra Tasks

Ultra Tasks support two types of pipeline designs, known as:
  • Low latency:
  • Headless Ultra

Low latency Ultra

This design is for pipelines that use a request and response construct.

  • One unconnected input view
  • One or more unconnected output views
  • A FeedMaster is required in the Snaplex.

This common design acts as a data access layer for real-time web services, transforming a pipeline into a continuously running job. Documents are fed into the pipeline through a FeedMaster, which queues the documents. The pipeline processes these documents and returns responses via the FeedMaster.

A simple Ultra Task might include a Mapper Snap that maps incoming documents to a target field.

Note: This setup differs from Triggered Tasks, which have one input and one output.

Headless Ultra

This design is for pipelines that require continuous data flow.

  • No unconnected input view and no unconnected output view
  • A FeedMaster is not required.
  • This setup is typically used if the data source is a Snap such as JMS Consumer.

This design is often used in a listener-consumer construct. An Ultra Task with no unconnected views can continuously poll and consume documents from an endpoint without requiring a document feed from the FeedMaster. For example, a JMS-based polling Ultra Task pipeline might include a JMS Consumer Snap and a Snap that writes data to a destination, such as a File Write Snap.

For Headless Ultra, some experimentation and care is required to avoid unexpected issues. Multiple instances might run when the Snaplex state is in transition, even if the configured instances is only one. You need to make use of the facilities offered by the message queue, like acknowledgement, to avoid processing the same event multiple times.

The following lists the Snaps and Snap Packs supported in Headless Ultra:

Design Guidelines for low latency Ultra Tasks

Ultra Tasks function similarly to web service requests and follow an HTTP response architecture. Consider the following guidelines when designing a pipeline to be called by an Ultra Task:

Naming
  • Avoid spaces in pipeline names that are invoked as tasks.
Input View Type

An unconnected input view type must be either binary or a document:

  • Binary: HTTP headers are in the binary document's header.
  • Document: HTTP request headers are in the root of the document, and the request body is in the content field.

    • Header Keys are always seen/received in all lowercase.

    • For example, you can reference the User-Agent HTTP header in the input document as $['user-agent'].

In addition to the HTTP headers in the request, the following fields are added to the input document:

  • uri: The original URI of the request.
  • method: The HTTP request method.

  • query: The parsed version of the query string. The value of this field is an object whose fields correspond to query string parameters and a list of all the values for that parameter.

For example, the following query string: foo=bar&foo=baz&one=1 results in the following query object:
{
"foo" : ["bar", "baz"],
"one": ["1"]
}
  • task_name: The name of the task.
  • path_info: The part of the path after the Task URL.
  • server_ip: The IP address of the FeedMaster that received the request.
  • server_port: The TCP port of the FeedMaster that received the request.
  • client_ip: The IP address of the client that sent the request.
  • client_port: The TCP port of the client that sent the request.
Output view type

The unlinked output view type can be binary or a document as well:

  • If the view type is a document:
    • The output document is JSON-encoded and sent back to the client as the response body.
    • To customize the response header and HTTP response code:
      • Map the response body to the $content field of the output.
      • Map the custom header fields to the root $ in the output.
      • Map the custom HTTP response code to the $status field of the output.
    • The other fields in the output document are treated as HTTP response headers to send back to the client.
    • If a content field is not present, the entire document is JSON-encoded and used as the response body send back to the client.
  • If the view type is binary data, the binary file's header is sent back to the client as HTTP response headers. The body of the binary file is directly streamed back to the client.
  • SnapLogic Platform expects exactly one output document for every input.
  • If the output is not sent, then the original HTTP request sent to the FeedMaster hangs, eventually reaching a timeout.
  • If more than one output document is generated, then SnapLogic Platform only sends the first one back as the response to the original HTTP request. This behavior is different from a Triggered Task, where all of the documents sent to the unlinked output are sent back in the response.
  • Error views are implicitly added to all Snaps when the pipeline is executed in Ultra mode, regardless of how the Snaps are configured.