Retrieve lineage in OpenLineage format

GET https://{controlplane_path}/api/2/{env_org}/rest/metadata/lineages/{pipeline_id}?format=OPENLINEAGE

Retrieve data lineage for all pipelines or for a specific pipeline that executed in this environment. The response contains JSON objects in the OpenLineage specification format. When you specify a pipeline ID, the results include historical changes such as when new Snaps were added.

Prerequisites

  • Environment (Org) admin permissions

Path parameters

Parameter Description Required
env_org The ID for the environment available from Admin Manager on the Environment page. Yes
pipeline_id Optional. Returns lineage for the specified pipeline. To find the ID, open a pipeline in Designer and copy the pipe_snode value from the URL. You might need to validate the pipeline before the pipeline contains the value. No

Query parameters

Parameter Description Required
start_ts An integer specifying the start time. No
end_ts An integer specifying the end time. No
page An integer specifying the page of results to return. No
format The value OPENLINEAGE returns results in OpenLineage format. Yes

Response

A successful response includes a Code 200 OK where:
  • size: The number of lineage objects returned.
  • page: The current page of results.
  • content: Each JSON object contains the data lineage for a pipeline in the OpenLineage Object Model.
{
  "size": 0,
  "page": 0,
  "content": [
    json1, json2, json3
   ]
}

Example

Assuming a request as follows with an Organization ID of abcdef123456789012345678 for an environment on the global elastic control plane: https://elastic.snaplogic.com/api/2/abcdef123456789012345678/rest/metadata/lineages/677777ff7170252eaef5cf7?format=OPENLINEAGE

Note in the example results:

  • The producer field includes the pipeline ID.
  • The producer and _producer fields provide a clickable link to open the pipeline in Designer.
  • The parent facet provides information about the pipeline run.

 {
            "eventTime": "2024-10-31T22:26:44.909Z",
            "producer": "https://elastic.snaplogic.com/sl/designer.html?
                         #pipe_snode=677777ff7770252eaef5cf7",
            "schemaURL": "https://openlineage.io/spec/1-0-0/OpenLineage.json#/definitions/RunEvent",
            "eventType": "COMPLETE",
            "run": {
                "runId": "66aa8584-5c3d-3547-a6c7-d5ea5a51b01e",
                "facets": {
                    "parent": {
                        "_producer": "https://elastic.snaplogicdev.com/sl/designer.html
                                      ?#pipe_snode=6724041ff7170252eaef5cf7",
                        "run": {
                            "runId": "5b746bcb-3ec8-460f-8f25-2995ed135119"
                        },
                        "job": {
                            "namespace": "SnapLogic",
                            "name": "jsonRefMapper.slp_20241031222635538235"

                        },
                        "_schemaURL": "https://openlineage.io/spec/facets/1-0-0/ParentRunFacet.json"
                    }
                }
            },
            "job": {
                "namespace": "SnapLogic",
                "name": "jsonRefMapper.slp_20241031222635538235":
                        GroupRows:001ac7e5-63c5-4ff9-bb86-73239ef5de37"
            },
            "inputs": [],
            "outputs": [
                {
                    "facets": {
                        "schema": {
                            "_producer": "https://elastic.snaplogicdev.com/sl/designer.html?
                                          #pipe_snode=6724041ff7770252eaef5cf7",
                            "fields": [
                                {
                                    "name": "rows",
                                    "type": "VARCHAR",
                                    "description": ""
                                }
                            ],
                            "_schemaURL": "https://openlineage.io/spec/facets/
                                           1-0-0/SchemaDatasetFacet.json"
                        },
                        "columnLineage": {
                            "_producer": "https://elastic.snaplogicdev.com/sl/designer.html
                                          ?#pipe_snode=6724041ff7770252eaef5cf7",
                            "fields": {
                                "rows": {
                                    "inputFields": [],
                                    "transformationType": "IDENTITY"
                                }
                            },
                            "_schemaURL": "https://openlineage.io/spec/facets/
                                           1-0-1/ColumnLineageDatasetFacet.json"
                        }
                    },
                    "namespace": "Snaplogic",
                    "name": "Virtual_DB.Virtual_Schema.GroupRows:001ac"
                }
            ]
        },