Create Cache Pipelines

Create Cache Pipelines to reference data from a main pipeline.

Cache pipelines enable you to run pipelines that refer to information from a main pipeline.

Prerequisites:

  • Cache pipelines must have the following input and output views:
    • 0 unconnected input views (Binary or Document)
    • 1 unconnected Document output views
    • 0 unconnected Binary output views
  • The output document must have a flattened document structure. None of the values for primary keys can have objects or arrays as part of their structure.
Important: If a document contains any value that is an array or object, it fails to process, which causes the pipeline to fail. If a cache pipeline fails to run, it will result in the preparation of the main pipeline failing (because it specifies the cache pipelines).
Build a Cache pipeline
  1. Open Designer and create a new pipeline. This pipeline generates data that has to be referenced in the main pipeline.
  2. Use Snaps to move data to the cache. Ensure the following:
    • The final output document needs to have a flattened structure.
    • The pipeline does not have an open input view. You can include a Mapper Snap to restrict the data in your cache or apply transformations based on specific requirements.
  3. Click Validate to validate the pipeline.
  4. Click Data Preview to open the Data Preview to verify the data that is stored in the cache.
Design the main pipeline
  1. Click your user name on the top right corner of the Designer tab.
  2. In the dropdown list, click User Settings.
  3. In the Settings dialog, set the Preview Document Count. You can choose one of the options (1, 2, 10, 20, 50, 100, 200, 500, 1000, 1500, 2000) from the dropdown. This value sets the limit for records allowed in the expression language lookup during validation.
  4. Click Save.
  5. Create or import a pipeline onto the Designer canvas.
  6. Click Edit Pipeline to open the Edit Pipeline dialog.
  7. Fill in the required details in the Edit Pipeline dialog:
    • Label: Enter a name for your main pipeline. We recommend that you choose a label that's easily recognizable as the pipeline.
    • Project: Confirm the correct project for the pipeline. If you need to change the Project, refer to Moving a Pipeline.
    • Resumable Mode: Leave deselected.
    • Error Pipeline: Leave empty.
    • Error Pipeline Parameters: Leave empty.
    • Parameters: Enter Pipeline Parameters.
    • Capture: Select to proliferate parameters downstream.
    • Expression Libraries: Add libraries as needed.
  8. In Cache Pipelines properties, click Add to enter the value in the Alias and the Pipeline properties to run for the data stored in that alias. These values run for the data stored in that alias, serving as the first argument in the Cache Pipelines expression language functions as a reference.
  9. Click Save.

Main pipelines reference the data in the cache pipeline. You can reference multiple main pipelines.