CSV file transformations

A data pipeline using the CSV file endpoint doesn't save the file names after the data loads to the target. However, AutoSync does save any transformation configuration. You can use the same data pipeline to upload one or more files with the same name, and AutoSync applies any previously configured transformations. This is true whether you use the Upload CSV file link on the data pipeline card, or edit it — when you select files with the same name.

For example, you might create a data pipeline named CSV-employees to upload data from an employees.csv file to Snowflake. This file lists phone numbers that shouldn't be exposed in the Snowflake table. When creating the data pipeline, remove the Phone column in the Filter columns dialog to prevent AutoSync from loading it. Later, the HR department hires more people and adds them to the employees.csv file.

You can add the new hires to the target table in one of the following ways:
  • From the CSV-employees data pipeline card:
    1. Click Upload CSV file.
    2. Select the updated employees.csv file.
    3. Click Save to upload the data without the phone numbers.
  • Double-click the CSV-employees data pipeline to edit it:
    1. In the Endpoints tab, select the updated employees.csv file.
    2. Click the Transformations tab and AutoSync prompts you to refresh the preview.
    3. Click the Refresh link in the message.
    4. The Removed columns list still contains the Phone column.
    5. Click Save to load the updated file without the phone numbers.