Tables

SnapLogic’s Data Catalog Services provides the capability to store metadata from pipelines into Table Assets and the functionality to query information about the actual source data without having to know where it is stored.

Tables Tab

The Table Asset helps you organize metadata by logically abstracting data stored in external storage systems, such as AWS S3. Over time, as data format, location, and schema change with new data passing through pipelines, this abstraction becomes useful.

  • Partition Keys: Your source tables are divided into partitions. You can enter these partition keys into the Table Asset, enabling users to work with the Table abstraction without needing to know the physical storage location of the data files.
  • Metadata Storage: Store metadata from your Pipelines by entering partition keys and schema information into the Catalog Snaps.
  • Catalog Snaps:
    • Catalog Insert and Catalog Writer Snaps populate a table with partitions.
    • Catalog Delete Snap removes them.

This functionality allows you to create, query, and delete Tables directly in SnapLogic, making data management more efficient and user-friendly.

Create table in Classic Manager

To create a new Table Asset directly in Classic Manager:

  1. Go to the Project folder in which you want to create the Table Asset.
  2. Click the the add icon ( ) icon on the right side of the menu bar, and click Table; the Create Table form is displayed.
  3. Enter the name of the Table in the Label field and the names of the Partition Keys. If there are multiple partition keys, enter each of them in a separate row.
  4. Click Save. The table is created and can be viewed from the Tables tab.

Query metadata and schema information in table partition

To query partitions in a Table Asset:

  1. In SnapLogic Manager, go to the Project where the Table Asset resides, then click the target Table. If you have several assets in the project, you can either search for the table or click the Table tab to filter out the tables and select the target table from there.
  2. Select a Partition Key from the Partition dropdown menu.
  3. Select an Operator from the dropdown menu (=, !=, =>, >, >=,<, <=,like), which are identical to those in the Catalog Query Snap.

  4. Enter the keyword in the Search Term field and click Go. The matching partitions are displayed. If you click Go when the fields under Search Partitions are empty, then all partitions are displayed.

    Data Catalog
  5. To display schema information for the entire Table, click Show Table Schema and a union of all schemas is displayed . Alternatively, you can upload a new schema for the Table through the displayed pop-up.
  6. For further analysis, select a partition:
    • To browse or search for Key and Value pairs, click the Metadata icon.
    • To search for or filter out specific names and attributes in the Schema, click the Schema icon.
    • To delete the partition, click the the delete icon ( )icon.

Delete table asset in Classic Manager

You can only delete a Table Asset in SnapLogic Manager.

To delete a Table:

  1. Go to the Project folder in which the target Table Asset resides, and click the Table tab.
  2. Select the target Table, then click the the delete icon ( ) icon. A pop-up prompts you to confirm the deletion.