Groundplexes and pipeline dependency on the control plane

Interactions between the SnapLogic control plane and data plane

This article describes interactions between the SnapLogic control plane and data plane and which types of pipeline assets require communication with the control plane. Although nodes communicate within a Snaplex, Snaplexes do not directly communicate with each other, regardless of whether the Snaplex is a Cloudplexes (SnapLogic managed) or Groundplex (self-managed). Because Snaplexes only communicate through the control plane, understanding the dependencies between the control plane and a Groundplex is helpful for implementation planning, pipeline design considerations, and troubleshooting.

Guidelines for reducing dependency on the control plane

Under certain conditions, pipelines can continue to process documents for some hours when a Groundplex node cannot communicate with the control plane. However, to reduce dependency on the control plane further, we recommend the following practices when building pipelines:

Avoid references to files uploaded to Manager in pipelines
Avoid files directly stored on customer-managed cloud storage like Amazon S3.
Avoid using dynamic references for assets such as accounts; instead, use static references. Statically referenced assets (such as accounts and child pipelines) are cached on the Snaplex.

The following types of pipelines run without access to the control plane using these guidelines:

Pipelines that process documents through an Ultra Task, which is already running, using the Low-latency Feed (Request and Response) design.
Standard-mode pipelines that are already started or an Ultra Tasks that use the Always-On (Continuous Data Flow) design.

Dependencies between Groundplex and control plane

The following table lists the dependencies between a Groundplex and the control plane for pipelines invoked as Ultra Tasks, Triggered Tasks, Scheduled Tasks, and child pipelines.


Type of Integration	Asset	Dependency (Yes/No)	Details
A running Ultra Task without Child pipelines	Accounts	No	No, if the reference is static. Yes, if the reference is evaluation as an expression or through OAuth. If OAuth is the authentication, then set a longer refresh token duration.
	Expression Library	No	No, if the reference is static. Yes, if the reference is evaluation as an expression or through OAuth.
	Pipelines	No	No, if the reference is static. Yes, if the reference is evaluation as an expression or through OAuth.
	File Read from Manager	Yes	Do not use files in Manager as file source.
	File Write from Manager	Yes	Do not use SLDB as file destination.
	Metadata Snap Pack	Yes	None
Ultra Task without child pipelines - Initializing	Accounts	Yes	No, if the reference is static. Yes, if the reference is evaluation as an expression or through OAuth. If OAuth is the authentication, then set a longer refresh token duration.
	Expression Library	Yes	No, if the reference is static. Yes, if the reference is evaluation as an expression or through OAuth.
	Pipelines	Yes	No, if the reference is static. Yes, if the reference is evaluation as an expression or through OAuth.
	SLDB File Read	Yes	Do not use files in Manager as file source.
	SLDB File Write	Yes	Do not use SLDB as file destination.
	Metadata Snap Pack	Yes	None
Ultra Task with child pipelines - initialization state or already running with child pipelines not running yet Scheduled Task (Initializing) Triggered Task - Cloud URL (Control Plane) Triggered Task - Alternative or Snaplex URL	Accounts	Yes	None
	Expression Library	Yes
	Pipelines	Yes
	SLDB File Read	Yes
	SLDB File Write	Yes
	Metadata Snap Pack	Yes
Scheduled Task (Already Running)	Accounts	No	Assumes the task is running for a very long time (such as days or weeks). Sample use case: Salesforce Subscriber Snap always running, or a File Poller Snap that polls indefinitely. However, you cannot view Pipeline Execution statistics in the dashboard if the control plane goes down.
	Expression Library	No
	Pipelines	No
	SLDB File Read	Yes
	SLDB File Write	Yes
	Metadata Snap Pack	Yes

Troubleshooting


Scenario	Work-around
Most Scheduled Tasks and Ultra Tasks when the pipeline is in the prepare state happen on the Snaplex. However, scheduled tasks still have a dependency on the control plane.	Change to an Ultra Task.
Statically defined OAuth accounts depend on the control plane - specifically the refresh token.	You can use client credentials in-memory to get around this which would depend on the Snaplex, instead of the Control Plane. The In-Memory OAuth2 Account relies on the client ID and secret to retrieve a new access token. One limitation is that it works only with the client credentials grant type and the services that do not invalidate the existing tokens after a refresh.