Best Practices for Ultra Tasks

The performance of pipeline invoked by an Ultra Task depends on the response times of the end-points to which the pipeline connects. An Ultra Task running a pipeline that contains many high latency endpoint Snaps can result in document congestion. They build all the way up through the upstream Snaps to the FeedMaster, until the FeedMaster queue can no longer hold the messages.

You can avoid this situation by either creating multiple instances of the Ultra Task or by using the pipeline Execute with the Reuse functionality enabled. Set the Pool Size field greater than one. Multiple instances ensure that even if one instance is slow, others are available to consume documents. Keep the FeedMaster queue flowing; a situation you can manage with the Max In-Flight setting on the Task form.

You can use a Router Snap in each pipeline instance to distribute the documents across multiple endpoint Snaps and improve the performance and add parallel processing capability to an instance. This functionality is in addition to the built-in parallel computation capability of a pipeline. Each Snap in a pipeline is processing a different document.

Scaling

Scaling can be attained by increasing the number of instances in an Ultra Task. The total number of instances required is a direct function of the expected response time: the resource utilization of the node (when a single instance of the Task is running) and the functional load on the Snaplex from other pipeline runs. When execution nodes are highly utilized, adding more execution nodes allows the instances of the Task to be distributed horizontally and scaled out across the Snaplex.

High availability Architecture

To avoid service disruption and enable high availability, use a load balancer with two FeedMasters and two execution nodes as the minimum architecture for Ultra Task setup. This architecture also avoids a single point of failure from the FeedMaster or execution node.

Monitor tasks in SnapLogic Manager

Each Task is listed in the Manager > Project > Task. To view the number of documents received or processed by each instance of the Task, a dropdown menu lists the details of the Task.

Use the Details menu to view the pipeline Execution stats and monitor documents received by each Snap in the pipeline instance. Since the detailed view only shows documents received by a specific instance of the Task, you might need a more advanced monitoring plan for multiple instances. This plan should provide an overview of all documents, queues, and subscribers.

Document lineage hierarchy

Every incoming document in an Ultra Task must maintain Snap lineage across the pipeline. This operation allows the FeedMaster to use the correlation ID to tie the request to the response. To maintain the lineage and identity of a static web service or file, use a Join Snap with a static key to link the static information to each request document. The following example shown demonstrates the usage of a Join Snap with a static configuration file for parameter mapping.

Child Pipelines in Ultra Tasks

You can also call child (nested) pipelines through Ultra Tasks by using the Pipeline Execute Snap. However, you should apply error handling to child pipelines so that errors occurring in any Snaps can be returned to the parent pipeline. You can add error views to the crucial Snaps in the parent pipeline, then introduce the error views to the Rest Get Snap calling the child pipeline.

Usage guidelines and limitations

Ultra tasks have the following limitations:
  • Snaps that wait until something is completed, such as Aggregate, Sort or Join, cannot be used in pipelines invoked by Ultra Tasks.
  • DB Snaps that write to the database must have the batch size in the account set to one; otherwise, they do not see the effect of the writes.
  • For the JSON Formatter Snap, select the 'Format each document' option.
  • For the XML Formatter Snap, clear the Root Element field.
  • Script and Execute Script Snaps need to pass the original document to the write() method for the output view. The following example shows the body of the loop in a JavaScript:
    // Read the next document, wrap it in a map and write out the wrapper
   var doc = this.input.next();
   var wrapper = new java.util.HashMap();
   wrapper.put("original", doc);
   this.output.write(wrapper);
The last line should look like:
   this.output.write(doc, wrapper);
  • The original document is needed so that lineage can be maintained.
  • If a pipeline called by an Ultra Task has an unlinked input view, then it should also have one or more unlinked output views as well.
  • By default, if the first run of an Ultra Task fails, SnapLogic attempts to run it for a total of five times. However, you can configure the number of times you want Ultra Tasks from a specific Snaplex to run by configuring the maximum number of allowed retries for the pipeline. To do so, modify the ultra.max_redelivery_count parameter in Global Properties in Snaplex Update to indicate the number of times you want a failed pipeline to run.
  • The maximum payload per request for an Ultra Task is based on the following calculation: 60% of the Java Virtual Machine heap size divided by 16. For example, if the JVM heap size is 10 GB, then the maximum size for the request payload is 375 MB per request (0.6 * 10 GB / 16).
  • You cannot change pipeline parameters at run time. These parameters are specified in the pipeline itself or when the Ultra Task is created. They are set at pipeline startup, which only happens on initiation, not per document.
Note: As a workaround, the Snaps with limitations may be used in a child pipeline and invoked locally using the Pipeline Execute Snap. Since this does not require communicating with the Control Plane, the local invocation can be fast.