AlloyDB Bulk Load

Overview

You can use this Snap to bulk load data into AlloyDB tables. If the target table does not exist in AlloyDB, you can create a table. This Snap supports both binary and document inputs. You can configure the input view in the Views tab.



Limitations

If the target table does not exist and the input is binary, the Snap does not fetch the table's schema from the input. The table's schema must be passed to the Snap through the second input view.

Behavior changes

  • The following are the behavior changes caused by the JDBC driver upgrade for PostgreSQL Snap Pack and Google AlloyDB Snap Pack
  • Some error messages will now include the data type for certain values, for example:

    • Old: "...where (badId = 1) was aborted:..."

    • New: 
      "...where (badId = ('1'::numeric)) was
    aborted:..."
  • There are some minor text changes to a few of the error messages, for example:
    • Old: 
      "Hint: No operator matches the given name and argument
      types."
    • New: 
      "Hint: No operator matches the given name and argument
    type(s)."

Snap views

View Description Examples of upstream and downstream Snaps
Input

The data to be loaded into the AlloyDB table. This Snap has one document input view by default.

A second view can be added for table metadata as a binary or document stream so that the target absent table can be created in the database with a schema similar to the source table. This schema is usually from the output of a database Select Snap.
Output The Snap outputs one document specifying the status, with the records count that are being inserted into the table. Any error occurring during the process is routed to the error view.
Error

Error handling is a generic way to handle errors without losing data or failing the Snap execution. You can handle the errors that the Snap might encounter when running the pipeline by choosing one of the following options from the When errors occur list under the Views tab. The available options are:

  • Stop Pipeline Execution Stops the current pipeline execution when an error occurs.
  • Discard Error Data and Continue Ignores the error, discards that record, and continues with the remaining records.
  • Route Error Data to Error View Routes the error data to an error view without stopping the Snap execution.

Learn more about Error handling in Pipelines.

Snap settings

Legend:
  • Expression icon (): Allows using pipeline parameters to set field values dynamically (if enabled). SnapLogic Expressions are not supported. If disabled, you can provide a static value.
  • SnapGPT (): Generates SnapLogic Expressions based on natural language using SnapGPT. Learn more.
  • Suggestion icon (): Populates a list of values dynamically based on your Snap configuration. You can select only one attribute at a time using the icon. Type into the field if it supports a comma-separated list of values.
  • Upload : Uploads files. Learn more.
Learn more about the icons in the Snap settings dialog.
Field / Field set Type Description
Label String

Required. Specify a unique name for the Snap. Modify this to be more appropriate, especially if more than one of the same Snaps is in the pipeline.

Default value: AlloyDB Bulk Load

Example: Load Employee Data
Schema String/Expression/ Suggestion

Specify the database schema name in AlloyDB. If you do not specify a schema, then the suggestion for the Table name retrieves all table names of all schemas.

Default value: N/A

Example: Employee
Table name String/Expression/ Suggestion Required. Specify the AlloyDB table on which to run the bulk load operation.

Default value: N/A

Example: Employee.PersonnelInformation
Create table if not present Checkbox Select this checkbox to automatically create the target table if it does not exist.

  • If a second input view is configured for the Snap and it contains a document with schema (metadata) of the source table, the Snap creates the new (target) table using the same schema (metadata). However, if the schema comes from a different database, the Snap might fail with Unable to create table: "<table_name>" error due to data type incompatibility.

  • In the absence of a second input view (the schema/metadata document), the Snap creates a table based on the data types of the columns generated from the first row of the input document (first input view).

Warning: A newly created table is not visible to subsequent database Snaps during runtime validation. If you want to immediately use the newly updated data you must use a child Pipeline that is invoked through a Pipeline Execute Snap.

Default status: Deselected
Columns Use this field set to define the columns to be loaded/copied. If you do not configure this field set, then all columns in the input data stream are loaded into the target table. You can also set how you want the Snap to handle null values for each column.
Warning:
  • Column order is critical when using the binary input view.

  • You can select both Force Non-Null and Force Null simultaneously. In such cases, the Snap converts quoted null strings to null values and unquoted null strings to empty strings.
Column String/Expression/ Suggestion Required. Specify the name of the column/field. This is a suggestible field and lists all field names in the input.

Default value: N/A

Example: emp_name
Force Not-Null Checkbox Select this checkbox to not match the selected column's value against the null string. The Snap reads empty null strings as zero-length strings rather than nulls, even when they are not quoted.

Default status: Deselected
Force Null Checkbox Select this checkbox to match the selected column value against the null string, even if it has been quoted. If a match is found, the field value is set to NULL.

By default, when the null string is empty, the Snap converts a quoted empty string into NULL.

Default status: Deselected
Format Dropdown list/Expression Required. Choose the data format to be written. This field is applicable only when using the binary input view.

  • Text - tab-separated-values (default)

  • CSV - comma-separated-values

  • Binary - AlloyDB specific format

Default value: Text

Example: CSV
Header Provided Checkbox Select this checkbox to include the input data as a header. Applicable only when using the binary input view and CSV option in the Format field.

Default status: Deselected
Encoding Dropdown list/Expression Choose one of the following encodings to use. This is limited to the encodings supported by the AlloyDB server:

Encodings:

ABC

ALT

BIG5

EUC_CN

EUC_JIS_2004

EUC_JP

EUC_KR

EUC_TW

GB18030

GBK

ISO88591

ISO885910

ISO885913

ISO885914

ISO885915

ISO885916

ISO88592

ISO88593

ISO88594

ISO88599

ISO_8859_5

ISO_8859_6

ISO_8859_7

ISO_8859_8

ISO_8859_8

JOHAB

KOI8

KOI8R

KOI8U

LATIN1

LATIN2

LATIN3

LATIN4

LATIN5

LATIN7

LATIN9

Mskanji

SHIFT_JIS_2004

SJIS

SQL_ASCII

ShiftJIS

TCVN

TCVN5712

UHC

UTF8

Unicode

VSCII

WIN

WIN1250

WIN1251

WIN1252

WIN1253

WIN1254

WIN1255

WIN1256

WIN1257

WIN1258

WIN866

WIN874

WIN932

WIN936

WIN949

WIN950

Windows932

Windows936

Windows949

Windows950

Default value: UTF8

Example: Unicode
Snap execution Dropdown list
Choose one of the three modes in which the Snap executes. Available options are:
  • Validate & Execute: Performs limited execution of the Snap and generates a data preview during pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during pipeline runtime.
  • Execute only: Performs full execution of the Snap during pipeline execution without generating preview data.
  • Disabled: Disables the Snap and all Snaps that are downstream from it.

Default value: Execute only

Example: Validate & Execute