Create a data pipeline

How to create an AutoSync data pipeline

To create an AutoSync data pipeline from the dashboard, click Create a new data pipeline:


AutoSync dashboard

In the wizard, the first screens ask about the purpose of the data pipeline, where to load the data, and which sources to use. You can select up to five sources for one data pipeline. AutoSync loads the data from all sources into the destination schema.

After you select endpoints, if a Groundplex is available, the wizard lets you choose whether to run this data pipeline on the AutoSync Snaplex or a Groundplex. AutoSync can connect to cloud endpoints with either the AutoSync Snaplex or a Groundplex. Some configuration is required to connect to on-premises endpoints.

As you work through the wizard, a description builds dynamically in the right side panel. The description documents your choices and makes it easy to share the pipeline's purpose with others. For example:


Dynamic data pipeline description

When you add endpoint credentials, a link on the icon indicates the connection status. For example, the green link in the screenshot shows that AutoSync successfully connected to Snowflake:


Connection status

Tip: Endpoint status indicators do not always update immediately. For example, if you successfully connected to an endpoint and then go back and enter invalid values, the status does not update until AutoSync attempts to reconnect to that endpoint.

Selecting tables

Table names must be unique in the destination schema. If you select tables or objects with the same name from multiple sources, AutoSync suggests unique names for the destination tables by appending the source type to the table name. You can accept the suggested names or enter your own.

If the source-target combination you select supports incremental load or SCD2, the table choice also has a field for a timestamp column:
Choose a column with a timestamp

To support incremental load, a timestamp column must:
  • Be one of the following types:
    • datetime
    • timestamp
  • Not be nullable.
  • Not contain null values.

Select transformations

The wizard transformation screens give you the option to transform data before loading it. Learn more about the available transformations.

Schedule synchronization

At the end of the wizard, you can schedule synchronization, save the data pipeline without running it, or run it immediately. After saving, AutoSync adds the data pipeline summary card to the dashboard. Environment admins (Org admins) can view all data pipelines in the Environment from the dashboard. Non-admin users can view data pipelines:

  • Created by them
  • Shared with them
  • When ownership is transferred to them