Best Practices for Ultra Pipeline Tasks
Performance of an Ultra Pipeline largely depends on the response times of the end-system applications to which the Task connects. An Ultra Pipeline containing a large number of high latency endpoint Snaps can result in a congestion of documents. They build all the way up through the upstream Snaps to the FeedMaster, until the FeedMaster queue can no longer hold the messages.
You can avoid this situation by either creating multiple instances of the Ultra Pipeline or by using the pipeline Execute with the Reuse functionality enabled. Set the Pool Size field greater than one. Multiple instances of an Ultra Pipeline ensures that even if one instance is slow, others are available to consume documents. Keep the FeedMaster queue flowing; a situation you can manage with the Max In-Flight setting on the Task form.
You can use a Router Snap in each pipeline instance to distribute the documents across multiple endpoint Snaps and improve the performance and add parallel processing capability to an instance. This functionality is in addition to the built-in parallel computation capability of a pipeline. Each Snap in a pipeline is processing a different document.
Scaling
Scaling can be attained by increasing the number of instances in an Ultra Pipeline. The total number of instances required for an Ultra Pipeline is a direct function of the expected response time: the resource utilization of the node (when a single instance of the Task is running) and the functional load on the Snaplex from other pipeline runs. When execution nodes are highly utilized, adding more execution nodes allows the instances of the Task to be distributed horizontally and scaled out across the Snaplex.
High availability Architecture
To avoid service disruption and enable high availability, use a load balancer with two FeedMasters and two execution nodes as the minimum architecture for Ultra Pipeline setup. This architecture also avoids a single point of failure from the FeedMaster or execution node.
Monitor tasks in SnapLogic Manager
Each Task is listed in the
. To view the number of documents received or processed by each instance of the Task, a dropdown menu lists the details of the Task.Use the Details menu to view the pipeline Execution stats and monitor documents received by each Snap in the Ultra pipeline instance. Since the detailed view only shows documents received by a specific instance of the Task, you might need a more advanced monitoring plan for multiple instances. This plan should provide an overview of all documents, queues, and subscribers.
Document lineage hierarchy
Every incoming document in an Ultra Pipeline Snap must maintain its lineage across the pipeline. This operation allows the FeedMaster to use the correlation ID to tie the request to the response. To maintain the lineage and identity of a static web service or file, use a Join Snap with a static key to link the static information to each request document. The following example shown demonstrates the usage of a Join Snap with a static configuration file for parameter mapping.
Child Pipelines in parent Ultra Pipelines
You can also call child (nested) pipelines through Tasks from Ultra Pipelines by using the Pipeline Execute Snap. However, you should apply error handling to child Ultra Pipelines so that errors occurring in any Snaps can be returned to the parent pipeline. You can add error views to the crucial Snaps in the parent pipeline, then introduce the error views to the Rest Get Snap calling the child Ultra Pipeline.
Usage guidelines and limitations
- Snaps that wait until something is completed, such as Aggregate, Sort or Join, cannot be used in Tasks.
- DB Snaps that write to the database must have the batch size in the account set to one; otherwise, they do not see the effect of the writes.
- For the JSON Formatter Snap, select the 'Format each document' option.
- For the XML Formatter Snap, clear the Root Element field.
- Script and Execute Script Snaps need to pass the original document to the write() method for the output view. The following example shows the body of the loop in a JavaScript:
// Read the next document, wrap it in a map and write out the wrapper var doc = this.input.next(); var wrapper = new java.util.HashMap(); wrapper.put("original", doc); this.output.write(wrapper); The last line should look like: this.output.write(doc, wrapper);
- The original document is needed so that lineage can be maintained.
- If an Ultra Pipeline has an unlinked input view, then it should also have one or more unlinked output views as well.
- By default, if the first run of an Ultra Pipeline fails, SnapLogic attempts to run it for a total of five times. However, you can configure the number of times you want Ultra Pipelines from a specific Snaplex to run by configuring the maximum number of allowed retries for the pipeline. To do so, modify the
ultra.max_redelivery_count
parameter in Global Properties in Snaplex Update to indicate the number of times you want a failed Ultra Pipeline to run. - The maximum payload per request for an Ultra Task is based on the following calculation: 60% of the Java Virtual Machine heap size divided by 16. For example, if the JVM heap size is 10 GB, then the maximum size for the request payload is 375 MB per request
(0.6 * 10 GB / 16)
. - You cannot change pipeline parameters at run time. These parameters are specified in the pipeline itself or when the task is created. They are set at pipeline startup, which only happens on initiation, not per document.