Partition Refresh to S3 with Catalog Update

This Pipeline queries a partition in the table, deletes the partition, generates a JSON document, and writes it to S3, inserting the metadata into the Table Asset in Manager.



Download this Pipeline
  1. Catalog Query Snap: Queries the Table to find the target partition.


  2. Catalog Delete : Deletes the Table partition.


  3. JSON Generator Snap: Generates a JSON document.


  4. Parquet Writer Snap: Writes the JSON document to an S3 bucket.


  5. Catalog Insert Snap: Writes the metadata into the Table.


    Viewing the Table in SnapLogic Manager
    You can check the result of the Pipeline by viewing the metadata for that Table.

    		 You can also view the Table schema for the partition, or upload a new schema for that Table, which contains all the fields in your source Table.
To successfully reuse pipelines:
  1. Download and import the pipeline into SnapLogic.
  2. Configure Snap accounts as applicable.
  3. Provide pipeline parameters as applicable.