| Initialization |
- Receives pipeline execution request from the external trigger.
- Initiates the process by identifying the relevant pipeline based on the execution
request.
- Assigns specific pipeline tasks to Snaplex nodes based on their availability and
capabilities.
|
- Receives pipeline execution request from control plane.
- Allocates memory and processing resources to prepare the environment to perform the
data integration tasks.
|
| Preparation |
- Accesses pipeline metadata to retrieve the required preparatory information.
- Performs validation and authorization checks to make sure that execution adheres to
defined security and access controls.
|
- Establishes connections to relevant endpoints. This includes connecting to
databases, APIs, and file systems.
- Checks to validate that the environment is ready and all prerequisites are met.
- Communicates readiness and initialization status back to the control plane.
|
| Execution |
- Loads pipeline metadata into its runtime environment.
- Monitors the progress of pipeline execution.
- Captures relevant metrics for auditing, troubleshooting, and performance analysis.
- Based on the real-time communication from the snaplex nodes, manges errors either by
predefined error-handling logic or by stopping the pipeline when required.
|
- Executes the assigned tasks in parallel to optimize performance and handle large
volumes of data efficiently.
- Handles errors locally by trying to resolve them before sending them to the control
plane.
|
| Completion |
- Logs error messages and timestamps.
- Calculates total output of each Snap and transformed data generated by Snaplex
nodes.
- Performs post-execution cleanup activities, including release of resources and
closing connections.
- Provides a final status report with the status of the pipeline execution and relevant performance metrics.
|
- Releases allocated compute resources, closes open connections, and cleans up temporary files.
- Sends comprehensive metrics to the control plane for analysis and optimization.
- Archives detailed execution logs.
|
