BigQuery Table List

Use this Snap to list tables that are read from a BigQuery dataset.

Overview

You can use this Snap to list tables that are read from a BigQuery dataset.



Prerequisites

A valid Google BigQuery Account with the required permissions.

Limitations and Known Issues

None.

Supported Accounts

This Snap works with the following account types. For more information, see Configuring Google BigQuery Accounts.

Snap Views

Type Format Number of Views Examples of Upstream and Downstream Snaps Description
Input Document
  • Min: 0
  • Max: 1
  • Mapper
  • JSON Generator

This Snap has at most one document input view.

The Project ID and the Document ID.
Output Document
  • Min: 1
  • Max: 1
  • JSON Parser
  • File Writer

This Snap has exactly one document output view. The list of Table IDs along with their Project IDs, Dataset IDs, and Table type.
Learn more about Error handling.

Snap Settings

Note: Learn about the common controls in the Snap settings dialog.
Field/Field set Description
Label

String/Expression

Specify a unique name for the Snap. Modify this to be more specific, especially if you have more than one Snap of the same type in your Pipeline.

Default value: BigQuery Table List

Example: Babynames Table List
Project ID

String/Expression

Specify the project ID in which the dataset resides.

Default value: N/A

Example: Test-project-324543
Dataset ID

String/Expression

Specify the dataset ID of the destination.

Default value: N/A

Example: project-test-43433
Detailed Information

Checkbox

Select this checkbox to enable the Snap to access additional fields for displaying them in the output.

Default value: Deselected
Snap execution

Dropdown list
Choose one of the three modes in which the Snap executes. Available options are:
  • Validate & Execute. Performs limited execution of the Snap and generates a data preview during pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during pipeline runtime.
  • Execute only. Performs full execution of the Snap during pipeline execution without generating preview data.
  • Disabled. Disables the Snap and all Snaps that are downstream from it.

Default value: Validate & Execute

Example: Disabled

Implicit retries in BigQuery Snaps

The BigQuery Snaps handle all retriable BigQuery errors (BigQuery exception, IO exception, and Runtime exception) internally.

  • 429 (Too Many Requests):
    • Retry attempts: Maximum of 5 retries.
    • Delay Between Retries: Backoff strategy with jitter (random variation) is applied to prevent synchronized retries and reduce load.
  • 401 (Unauthorized):
    • Retry attempts: Maximum of 3 retries.
    • Delay Between Retries: Backoff strategy is applied.
    • Additional Actions: Reloads the BigQuery account on the retry event.
  • IOException and 500, 502, 503, 504 (Server Errors):
    • Retry attempts: Maximum of 3 retries.
    • Delay Between Retries: Backoff strategy is applied.