Watchdogs

In hard real-time industrial automation, a task that misses its execution deadline or hangs in an infinite loop can lead to hazardous machine behavior. Adapnex provides built-in software execution watchdogs on cyclic task groups to detect cycle overruns, scheduling stalls, and thread deadlocks.

Watchdog Mechanics

Every cyclic task group can monitor its execution timing against configured limits. When enabled, the watchdog tracks the elapsed duration of each update cycle across all registered tasks.

The watchdog monitors three timing conditions:

  1. Single-Cycle Overrun: An individual cycle takes longer than the configured watchdog_time.

  2. Cumulative Budget Overrun: A cycle fails to complete within the total execution budget (watchdog_sensitivity multiplied by watchdog_time).

  3. Omitted Cycle or Late Dispatch: A cycle is delayed in starting beyond the allowed dispatch tolerance.

Configuration

The execution watchdog is configured when creating a cyclic task group via CyclicTaskGroupConfig:

#include "adapnex.h"

void setup() {
    // Enable the watchdog with a 10ms deadline and a sensitivity of 3 cycles
    const auto group = Application::CreateCyclicTaskGroup({
        .period = 10ms,
        .watchdog_time = 10ms,
        .watchdog_sensitivity = 3,
    });
}

Parameters

The watchdog configuration parameters are defined in CyclicTaskGroupConfig:

  • watchdog_time: The execution deadline for an individual cycle. Setting either watchdog_time or watchdog_sensitivity enables the watchdog. If omitted, watchdog_time defaults to the cycle period.

  • watchdog_sensitivity: The number of consecutive overrun cycles tolerated before the watchdog trips. It also defines the multiplier for the single-cycle execution budget (sensitivity multiplied by watchdog_time). A sensitivity of 0 is treated as 1.

  • watchdog_handler: An optional callback invoked when the watchdog trips. If omitted, the runtime reports the fault and terminates the application immediately via std::abort().

The total execution budget for any individual cycle is watchdog_sensitivity multiplied by watchdog_time. A single cycle whose execution exceeds this budget trips the watchdog immediately.

Trip Reasons

When a watchdog trips, it constructs a WatchdogEvent describing the fault. The event provides a reason property of type WatchdogTripReason:

Trip Reason Description

kCycleDidNotComplete

A single cycle took longer than the total execution budget (sensitivity multiplied by watchdog_time), and the watchdog tripped while the cycle was still actively executing.

kCycleOverranBudget

A single cycle completed its execution, but the total measured duration exceeded the total execution budget (sensitivity multiplied by watchdog_time).

kConsecutiveOverruns

Consecutive cycles each exceeded the single-cycle watchdog_time without exceeding the total budget, and the overrun count reached watchdog_sensitivity.

kCycleOmitted

A cycle did not start within the allowed dispatch tolerance, indicating that the operating system thread was starved or delayed.

The WatchdogEvent structure also provides:

  • period: The configured cycle period of the task group.

  • watchdog_time: The watchdog time limit in effect.

  • watchdog_sensitivity: The sensitivity count in effect.

  • elapsed: The actual duration of the cycle, or how late the cycle started for omitted cycles.

  • Message(): A formatted human-readable diagnostic sentence summarizing the trip event and timing details.

Default Abort Behavior

By default, when the execution watchdog trips, it calls std::abort() to terminate the process immediately. It does not throw an exception, and task teardown (Task::TearDown()) does not run.

Understanding why the watchdog aborts is crucial:

  • Inability to Terminate Without Abort: A common pitfall is expecting the watchdog to trigger a graceful shutdown or throw a catchable C++ exception. However, when a task is stuck in an infinite loop, caught in a deadlock, or blocked in an uninterruptible system call, the thread can never return. Because the thread is hung, standard control flow, exception unwinding, and graceful shutdown loops cannot execute. Without calling std::abort(), the program would hang indefinitely and would never be able to terminate at all.

  • Corrupted State Integrity: When deadlines are severely missed, internal application state or memory invariants may already be compromised. Attempting to run normal cleanup code on corrupted state can cause secondary crashes, emit invalid hardware commands, or stall during teardown.

  • Automatic Recovery via Restart: Adapnex applications are designed to run under process supervision with automatic restart capability. When a process terminates abruptly via std::abort(), the supervisor restarts the application immediately. On restart, the fresh application process can detect the previous crash, perform hardware reconciliation and cleanup, and safely restore the machine to a known operational state.

Custom Handlers

You can register a custom watchdog_handler to execute diagnostic actions or emergency recovery logic when the watchdog trips:

#include "adapnex.h"
#include <iostream>

void EmergencyWatchdogHandler(const WatchdogEvent &event) {
    std::cerr << "CRITICAL FAULT: " << event.Message() << std::endl;

    switch (event.reason) {
        case WatchdogTripReason::kCycleDidNotComplete:
            std::cerr << "Execution hung or exceeded total budget. Aborting process." << std::endl;
            std::abort();

        case WatchdogTripReason::kCycleOverranBudget:
            std::cerr << "Single cycle overran total budget (" << event.elapsed.count() << "us). Aborting process." << std::endl;
            std::abort();

        case WatchdogTripReason::kConsecutiveOverruns:
            std::cerr << "Repeated consecutive overruns detected (" << event.watchdog_sensitivity << " times). Requesting shutdown." << std::endl;
            Application::Stop();
            break;

        case WatchdogTripReason::kCycleOmitted:
            std::cerr << "Cycle dispatch stalled by system scheduler. Requesting shutdown." << std::endl;
            Application::Stop();
            break;
    }
}

void setup() {
    const auto group = Application::CreateCyclicTaskGroup({
        .period = 5ms,
        .watchdog_time = 5ms,
        .watchdog_sensitivity = 2,
        .watchdog_handler = EmergencyWatchdogHandler,
    });
}

Handler Guidelines

When implementing a custom watchdog handler, observe the following considerations:

  • Thread Safety: The handler may be invoked from a separate watchdog monitoring thread while a task is still running. Any data accessed by the handler should be thread-safe.

  • No Task Context: The task group execution context is not active within the handler. Timers (TON, TOF, TP) cannot be used inside the callback.

  • Remaining Task Groups: When a watchdog trips, that specific task group stops running further cycles. Other task groups continue running unless explicitly stopped via Application::Stop().

  • Process Lifetime: If a custom handler returns without calling Application::Stop() or std::abort(), the process remains running while the tripped task group stays halted.