Deployment architecture for Ultra Tasks

Ultra Tasks require consideration and preparation of your network depending on the purpose of the pipelne. The following deployment architectures cover the most common scenarios for setting up your environment to run pipelines with an Ultra Task.

One component present in most configurations is the FeedMaster, which brokers the queues between the execution nodes and Clients. The FeedMaster receives HTTP requests and turns them into documents and sends them to nodes for processing. The response from the node sends it to the FeedMaster, which in turn returns it as a response to incoming requests.

No FeedMaster

Headless Ultra Tasks does not provide a REST API interface for the pipeline and therefore does not require a FeedMaster, so setting up this Snaplex is simpler than other configurations. This is just providing Always-On pipeline functionality without requiring the FeedMaster infrastructure.

The Ultra Task functionality ensures that N number of listener pipelines are always running.

Use this configuration if you already have a data source that provides a listener-type of interfaces, such as JMS or file poller. When you use a JMS listener in a standard pipeline execution, an individual instance of the pipeline just runs, waiting for input documents to arrive from the JMS queue. Each instance is individual. When you define the Ultra Task, you can ask for n number of instances to be instantiated across the Snaplex, which the SnapLogic Platform monitors and restarts automatically, ensuring that the full number of instances are continuously running.

One FeedMaster Node

This configuration allows you to build a low latency REST interface for communicating with any type of endpoint. It provides low latency and reliability as compared to a Triggered Task, which is dependent on the control plane.

  • The FeedMaster node installed in front of the Snaplex JCC node.
  • A FeedMaster manages the distribution of requests for the Snaplex JCC node, which does the data processing.
  • The HTTP request is turned into a document.
  • Plain HTTP or HTTPS requests can be fed into a pipeline through the Snaplex FeedMaster node.
  • This document is sent to the pipeline's unconnected input, which is turned into the HTTP response to the original request.

  • The FeedMaster uses the following ports:

    • 8084: The FeedMaster's HTTPS port.
    • 8089: The FeedMaster's embedded ActiveMQ broker TLS (SSL) port.
    Important: FeedMaster ports: The machine hosting the FeedMaster must have those ports available on the local firewall.

Standard configuration: Two FeedMasters and one load balancer

The following diagram provides an overview of the standard network architecture required for Ultra Tasks, including the communication between the various components with a focus on the network: ports.Standard network architecture of Ultra Tasks

Disaster recovery configuration: Four FeedMasters and two load balancers

Ultra Tasks support a disaster recovery configuration, where you use the Alias field to call the same pipeline across two tasks on different Snaplexes. This configuration requires implementation via your IT organization. You need to configure your load balancer or manage through your Organization's DNS configuration to recognize when your Snaplex is offline during a DR situation.

The following diagram provides an overview of the Disaster Recovery architecture for pipelines invoked by Ultra Tasks.Disaster Recovery architecture for Ultra Tasks.

Configuration on DNS Server

  • If you are deploying the Snaplex in an Active-Active configuration, set Route53 to support Active-Active pairs across the load balancers (East and West).
  • If you are deploying the Snaplex in an Active-Standby configuration, set Route53 across the load balancers (East and West).

Deployment of tasks

  • All Tasks deployed in the Eastern region should have the same unique prefix and suffix; likewise, all tasks deployed in the Western region should have the same unique prefix and suffix.
  • For example, the Task, task_name_TaskOrgNameEast, is deployed to UltraPlex East, where TaskOrgNameEast is a unique string that is present as the suffix for all Tasks deployed on UltraPlex East.
  • Similarly, the Task, task_name_TaskOrgNameWest, that is deployed to UltraPlex West, where TaskOrgNameWest is a unique string that is present as the suffix for all Tasks deployed on UltraPlex West.
  • Both Tasks must have the same alias.

Configuration guidelines for a load balancer

  • Load balancers require FeedMaster port information.
  • Configure an alias on both Tasks. The alias name is used for the Task URL.
  • Use the HealthZ URL https://<HOSTNAME>:8084/healthz to monitor communication between the load balancers and the FeedMasters. For details, see Deploying a FeedMaster Node.