Scheduled Task metrics use cases

Monitoring scenarios that show how to use the scheduled-task lifecycle metrics to see whether a scheduled task fired on time, fired at all, and prepared successfully — before a downstream failure or a customer report surfaces the problem.

Scheduled-task lifecycle metrics give you real-time insight into how every Scheduled Task behaves, from the instant it is due to the moment its pipeline takes over. They expose lateness (drift), missed executions (misfires), and pipeline-preparation outcomes that otherwise remain invisible until something downstream breaks or you investigate the logs.

These metrics use the plexnode.state.task.scheduled.* prefix, are tagged with task_path and snaplex, and are enabled by default. For the metric definitions and types, see the scheduled-task metrics in the Snaplex metrics reference. Like the other Snaplex metrics, they are emitted through Third-party Observability, so you can build the panels described here in any compatible tool (Datadog, Grafana, or a Prometheus stack).

Note: Scheduled-task metrics cover the preparation phase — everything up to the point the pipeline starts running. The execution phase itself (whether the run succeeded, failed, or timed out) is captured by the plexnode.pipeline.* metrics. See Pipeline lifecycle metrics.

Monitoring scenarios at a glance

The following table summarizes the common monitoring scenarios for Scheduled Tasks, the metrics that support each one, and what to watch for. The sections that follow explain how to use the metrics in each scenario.

Use case Scenario Metrics to watch What to look for
Schedule drift Time-critical tasks must fire on schedule. Median, 95th percentile, and max of plexnode.state.task.scheduled.drift.ms, tracked over time. A tail above P95 and a sustained upward drift trend.
Missed executions A skipped run must never go unnoticed. plexnode.state.task.scheduled.misfire (count and rate). Any non-zero value — a scheduled run was skipped because it could not start within its lateness grace period.
Preparation health A task must prepare its pipeline before it can run. plexnode.state.task.scheduled.prepare.success, plexnode.state.task.scheduled.prepare.failure, and plexnode.state.task.scheduled.prepare.duration.ms. Rising prepare failures, or prepare duration climbing (an upstream slowdown).
Run outcomes Recurring tasks must fire and succeed. plexnode.pipeline.execution.stop.total by outcome, filtered on root_invoker_type="scheduled" (a pipeline metric). Rising failed or stopped outcomes for scheduled runs.
Backlog and capacity The scheduler must keep pace with load. Node capacity metrics — plexnode.slots.leased versus plexnode.slots.max and plexnode.pipelines.active.total — read alongside drift and misfire trends. Slot utilization approaching the maximum, or queued Scheduled Tasks — scale before runs drop.

Schedule drift

Use plexnode.state.task.scheduled.drift.ms to see how late a task fires relative to its planned schedule. Because it is a histogram, plot the median, 95th percentile, and maximum over time, filtered by task_path, rather than a single average. A healthy task holds a low, flat drift; a tail that climbs above the P95 you expect, or a sustained upward trend, indicates the scheduler is falling behind for that task and its on-time SLA is at risk. Alert on the P95 (or max) crossing your SLA threshold rather than on any single late fire.

Missed executions

A misfire is a missed execution: the scheduled fire time plus a lateness grace period elapsed before the task could start, so the pipeline was never invoked for that occurrence and the run is recorded as failed. This is distinct from drift, which measures a run that started late but still ran. plexnode.state.task.scheduled.misfire is a meter, so you can track both the total count and the rate. Treat any non-zero misfire count for a critical task as actionable — it means a scheduled run was skipped. Alert on the misfire rate so that missed runs are surfaced immediately instead of going unnoticed until a customer reports missing data. For how the grace period is determined, see Misfires in Scheduled Tasks.

Preparation health

Before a scheduled task runs, its pipeline must be prepared. Track the ratio of plexnode.state.task.scheduled.prepare.failure to plexnode.state.task.scheduled.prepare.success to catch tasks that fire but fail to start — for example, because of a missing account, an unresolved dependency, or a validation error. Plot plexnode.state.task.scheduled.prepare.duration.ms (a histogram) over time: a climbing prepare duration is an early warning of an upstream slowdown, and it quantifies preparation errors that are otherwise buried in the logs.

Run outcomes

The scheduled-task metrics cover preparation, not the execution result. To monitor whether scheduled runs actually succeed, use the pipeline execution metric plexnode.pipeline.execution.stop.total grouped by its outcome tag (completed, failed, stopped) and filtered on root_invoker_type="scheduled". Watch for a rising share of failed or stopped outcomes. For the full worked scenario and an example query, see the scheduled-task use case in Pipeline metrics use cases.

Backlog and capacity

Scheduled Tasks compete for a node's execution slots. When a Snaplex is at capacity, a scheduled run can be delayed or, with Scheduled Task Queuing enabled, held in a queue until resources free up. There is no dedicated scheduled-task backlog metric; instead, gauge whether the scheduler is keeping pace by watching node capacity: compare plexnode.slots.leased against plexnode.slots.max and track plexnode.pipelines.active.total. Slot utilization that rides near the maximum, especially alongside rising drift or misfires, means the node is falling behind — scale out or rebalance load before runs start dropping. You can also view slots, active pipelines, and CPU, memory, and disk utilization per node on the Metrics tab of the node details panel in Monitor.

Limitations

  • Scheduled-task metrics report the preparation phase. Execution results (success, failure, timeout, throughput) come from the plexnode.pipeline.* metrics; see Pipeline lifecycle metrics.
  • Misfire rate and drift percentiles are derived in your monitoring tool from the misfire meter and the drift.ms histogram; they are not separate metrics.
  • There is no dedicated scheduled-task backlog metric. Gauge whether the scheduler is keeping pace by watching node capacity (slots and active pipelines) and drift and misfire trends, as described in Backlog and capacity.