Google BigQuery as a source

You can create a data pipeline that loads data from Google BigQuery to a destination. To provide the information SnapLogic AutoSync needs to connect to Google BigQuery, supply new credentials in the wizard or select saved credentials. The create credentials page explains how to create credentials in the AutoSync wizard.

To use the Incremental load type, you'll need to select a column that contains the last updated timestamp for each table. If no such column exists, you can:
  • Create it in the source first.
  • Use Full load instead. Full load drops the target tables and reloads them on synchronization. It does not require a timestamp column that tracks updates. AutoSync uses this load type by default if you do not select a timestamp column.

A timestamp column must be one of the following types:

  • datetime
  • timestamp

Supported Snap account types

Designer and Classic Manager provide multiple Snap account types for most endpoints and not all types are compatible with AutoSync. When you create or edit a data pipeline, the existing credentials list includes only compatible accounts.

Many Snap accounts support references to JAR files. If you create an account that references a JAR file for use in AutoSync, you must use an absolute path to the JAR file.

For BigQuery, AutoSync supports:

  • Google BigQuery OAuth2

Known limitations

  • When loading data from a Google BigQuery source to a Google BigQuery destination, the source schema must be different than the destination schema.
  • When using BigQuery as both a source and a target, the loaded data for some columns might be a different data type than they were in the source. This is because AutoSync maps data through a common data model.

Connection configuration

Google BigQuery properties include the following:

  • Credential label: A unique, meaningful name such as Sales-Shared-BigQuery. If a configuration with the same name exists, AutoSync displays an Asset conflict error message.
  • Authorize: This button authorizes AutoSync with your Google account.
  • Share: (Optional) Select a user group to share this configuration with. Environment admins (formerly Org admins) create user groups to share credentials. If you are a member of a user group, you can select it from the dropdown. You can also select the global shared folder, which shares the credentials with everyone in your Org.
  • Validate and save: After saving, AutoSync adds the configuration to the list of saved credentials.
  • Enter Project ID: Enter the ID of the BigQuery Project to load data into.
  • Select schema: AutoSync populates this list from the account. Choose the schema to use as the destination.
  • Location: (Optional) If you are using a single region dataset, you can select the region in the Location field. If you are using a multi-region dataset, you can leave this field blank.