> For the complete documentation index, see [llms.txt](https://docs.layeronecloud.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.layeronecloud.com/network-monitor/architecture/agent-health.md). # Agent health, recovery and paging The agent supervises its monitoring functions independently: capture workers, WAN probes, telemetry upload, configuration, inventory, the spool, and evidence handling. ## Agent health and automatic recovery The agent supervises its monitoring functions independently: capture workers, WAN probes, telemetry upload, configuration, inventory, the spool, and evidence handling. A function that fails or exits unexpectedly is retried with bounded backoff and resource reopening. Repeated local operation failures retain the wire state `failed`; retries continue so a repaired dependency can recover without a reinstall. The panel calls reachable impairments **Degraded**, not "self-healing failed." Keyed dependency waits use wire `recovering` without incrementing attempts; a successful unrelated operation cannot clear them. The attempt counter measures local failed operations scheduled for retry, not process restarts or repeated observations of another worker's status. A quiet interface is healthy when polling and counter reads keep working. A WAN target failing to answer is a network measurement, not a reason to restart host networking or change routes, bridges, firewall rules, or VMs. A separate health thread sends authenticated JSON directly to the control plane every 15 seconds. These reports bypass SQLite, the telemetry upload queue, and Redis ingestion. They include a reporter session ID, increasing sequence, and each function's state, bounded diagnostic message, and recovery attempt count. The server validates the installation identity and records receipt using its own clock; sequence replays and retired reporter sessions cannot extend liveness. An explicit retired-session response causes the live reporter to open a fresh session and sample current health again. The agent also advertises health support on configuration and telemetry requests, so the panel detects a reporter that never sends its first heartbeat. Telemetry receipt and processing progress have separate timestamps, so a working health callback cannot conceal a broken telemetry pipeline. The older queued health record remains as a periodic SQLite write-recovery check. Its unused spool counters stay at their protobuf defaults, avoiding a scan of every queued payload every 30 seconds. The direct diagnostic reports and WAN latency/loss probes keep their existing cadence. The reporter also services the systemd watchdog. Workers that stop making progress cause watchdog keepalives to stop, allowing systemd to restart the agent process. An operation still making bounded retry attempts can report its failure without repeatedly restarting the entire service. This second layer covers a stuck worker or reporter; the panel's independent deadlines cover a dead process, host, or callback path. The separate scanner service checks its own worker's progress. A stuck worker causes a nonzero scanner-process exit, allowing its installed systemd `Restart=on-failure` policy to restart it and reap the Nmap control group. Capture remains in the main agent process. Configuration lease waits and long scans with a responsive supervisor do not trigger this restart path. There is no Nmap runtime deadline; a stuck child whose supervisor still pulses is not covered by this detector. These mechanisms cannot repair an unavailable control plane, bad permissions, or a persistently failing disk; they retry or report the dependency instead. ## Agent reliability paging The panel and dispatcher share a three-state model: `running`, `degraded`, `offline`. Independent wire statuses and per-function diagnostics remain available in health details. Diagnosis is separate from alert eligibility: | Status / issue | Independent evidence and alert policy | | --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | | Running | Contact is fresh; supported agents must also confirm healthy functions, applied configuration, and fresh processing | | Degraded / `agent_degraded` | Reachable but functions, configuration, health reports, or telemetry are impaired; alert only after 900 continuous degraded seconds | | Offline / `agent_offline` | No authenticated contact for 60 seconds; immediately eligible, taking precedence over degradation | Missing health is detected after 60 seconds; stalled telemetry requires both current processing and queue-progress timestamps to be stale for 120 seconds. Ordinary backlog catch-up is not a stall. `health_degraded_since` is persisted under the Agent row lock by accepted health reports and alert evaluation, not by UI reads. Reason changes, restarts, or an intervening Offline state do not clear this clock before confirmed Running. One stable `agent_degraded` episode keeps changing causes from producing separate alerts. Recovery verification retains this same state and deadline: it cannot suppress an alert indefinitely if current telemetry never confirms recovery. Migration `0057_agent_degraded_since` adds the clock and normalizes existing silence settings to the fixed minute. Loss of contact takes precedence, then missing health, then failed recovery, then stalled telemetry. Both the agent table and alert list show the reason. For upgraded agents, closing an episode requires fresh healthy reports spanning at least 30 seconds and telemetry processed after that healthy period began. Each existing episode also requires 30 seconds of fresh reports after the panel first observes that its issue has cleared; a recurrence restarts this confirmation. A restart, a single good heartbeat, or a config poll alone cannot resolve it. Agents with no health support keep their existing offline detection and show an upgrade notice. Sensors that have never checked in, are revoked, or are locked out as duplicates are never paged. While an outage alert is open and unacknowledged, every enabled on-call number is called once per round on the configured interval. Twilio reads the sensor name, site, and failure reason, then gathers a single keypad digit. Pressing 1 stops the rounds; the alert stays open so the console still shows what is down. Once one recipient acknowledges, the dispatcher hangs up the calls still ringing on the other phones. Degraded agents use the SMS-only class: one combined message after the grace period, repeated hourly until acknowledged. They never call; disabling SMS leaves those alerts in the panel only. Existing legacy health-failure incident types remain readable during rollout. Texting accompanies the calls rather than replacing them, and carries none of the repetition: one message per recipient when the alert opens and one when the sensor recovers, so the thread stays a readable record while the phone does the waking. A recovery message goes only to people who were told about the outage. Replying `1` acknowledges the alert whose message is newest in that thread, and `ALL` acknowledges every open alert the sender was texted about, which is what a rack-wide outage needs. Twilio delivers replies to whatever the number itself is configured with rather than to a per-message URL, so acknowledging by text requires pointing that one webhook at the console. Incidents are opened and resolved whether or not Twilio is configured, and an event with the corresponding issue type opens alongside each one so failures reach the events feed operators already watch. Credentials decide who gets told, never whether the failure is recorded. The dispatcher is the only component that talks to Twilio. Call and message webhooks change state and answer immediately, so a slow or unreachable provider can never delay the acknowledgement a caller is waiting to hear. Both call rounds and message notifications are claimed with a conditional update on the alert row, so running more than one dispatcher replica cannot double-call or double-text. These checks depend on the control plane and dispatcher being available. Their health must also be monitored from outside this deployment, including the dispatcher heartbeat and the actual notification path; an application cannot reliably page about its own complete host or database outage. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.layeronecloud.com/network-monitor/architecture/agent-health.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.