> For the complete documentation index, see [llms.txt](https://docs.layeronecloud.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.layeronecloud.com/network-monitor/operations/alerts.md).

# Outage, DDoS and agent reliability alerts

A WAN that is measured down pages on its own, per uplink.

## WAN outage alerts <a href="#wan-outage-alerts" id="wan-outage-alerts"></a>

A WAN that is *measured* down pages on its own, per uplink. A sensor going silent already pages as an offline agent, and on a single-uplink node that is the same event; a second uplink breaks that equivalence, so each configured WAN link has its own incident. The link must stay measured down, no carrier or no Internet target reachable, for **WAN outage threshold** seconds (sixty by default) before anybody is called, so a single failed round cannot ring. An unverified link never pages: silence from the probes might be the reporting path, and the offline and telemetry-stalled alerts already cover that. Calls pause while a link cannot be measured and resume if it is measured down again. A measured recovery resolves the incident and, when texts are on, sends the recovery message with the outage length. The Events page mirrors each outage as a critical `wan_down` event.

## DDoS attack alerts <a href="#ddos-attack-alerts" id="ddos-attack-alerts"></a>

A detected attack large enough to hurt the site calls the on-call numbers. **Call on large DDoS attacks** arms it and **DDoS call threshold** sets the size in gigabits per second, five by default and adjustable between 0.1 and 1000; anything smaller is recorded on the DDoS page and rings nobody. Gigabits are the unit end to end, from the console field through the API, and the panel stores the exact bit rate it will compare a detection against, so a threshold of 7.5 pages at 7.6 Gbps and not at 7.4. The estimate compared against it is the peak bit rate of the correlated site incident, which is the largest rate any one observing sensor measured, never their sum: mirrored observations of one flood must not add up to an attack that did not happen.

The trigger is the peak rather than the current rate, for two reasons. A flood oscillating around the threshold would otherwise ring, clear, and ring again; and an attack that has been partly absorbed has not stopped, so the person holding the phone still needs it. The call names the peak and the current rate so neither is mistaken for the other.

Sensors watching one flood are already correlated into a single site incident, so one attack is one call however many of them see it. The incident is opened against the earliest-observing sensor that is still deployed, and moves to another observer if that one is retired mid-attack. Paging stops when the sensors stop reporting the flood for ninety seconds or it resolves; with texts on, that sends the recovery message. Detection already writes its own `ddos_detected` event, so this adds the call and no second entry in the feed.

Raising the threshold above a running attack's peak, or turning the switch off, closes the alert without announcing a recovery that did not happen.

## Agent reliability alerts <a href="#agent-reliability-alerts" id="agent-reliability-alerts"></a>

Two rules keep the diagnostic cause clear:

* **Working through a backlog is progress, not a stall.** A sensor catching up after an outage uploads batches that were generated hours ago, so nothing it sends can make processing look current. A worker records that it applied something for that sensor, and *telemetry not being processed* needs both that and the sensor's own processed timestamp to be stale. Before this, a correct recovery paged continuously for its whole duration.
* **Panel failures are identified as panel failures.** When storage is refusing writes or no telemetry workers are running, stalled telemetry can reflect the panel rather than the sensor. After the degraded grace period, the SMS-only alert includes the detected panel issue, where available; it does not claim a failed sensor restart. Check *Uploads refused*, the worker count in **Telemetry processing**, and `/healthz/`.

The agent has three general operational states:

| Status   | Meaning                                                                                                                        | Alert threshold                     |
| -------- | ------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------- |
| Running  | Reachable, with healthy monitored functions and current telemetry                                                              | None                                |
| Degraded | Reachable, but a function, configuration, health reporting, or telemetry path is impaired; also used while confirming recovery | 15 continuous minutes in this state |
| Offline  | No authenticated contact for one minute                                                                                        | 60 seconds without contact          |

Degradation is visible immediately after it is detected. The server persists its start time across changing reasons and agent restarts; switching from a scanner problem to an upload problem does not restart the grace period. Only verified Running clears it. Offline takes precedence and does not wait for the degraded timer. Locked-out, revoked, and never-enrolled agents retain their separate administrative states and are not paged.

Failed live uploads, unavailable upload servers, and unsuccessful configuration refreshes are dependency degradation, not proof that a recovery attempt failed. The sensor retains durable batches and keeps capture running on applied settings. Scanner permission expires without fresh configuration, safely deferring scans until it returns. Observing the same wait every five seconds does not increment the recovery-attempt count.

After its threshold, an offline or degraded episode opens an alert and an event in **Agents & Settings**. Short degradation remains in health details without creating an alert episode. Measured WAN outages and large DDoS attacks retain their own independent thresholds. Configure recipients and credentials there.

Alerts are delivered in two classes, and notifications combine within a class and never across it:

| Class    | Alerts                                                 | Delivery                                                                                                                                                           |
| -------- | ------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Outage   | offline sensor, WAN down, DDoS over the call threshold | One call to every on-call number, repeated every **Repeat every** seconds (60 by default) until acknowledged, plus one text when each opens and one when it clears |
| Degraded | reachable sensor with a sustained impairment           | No calls. One text listing affected agents and causes after 15 minutes, repeated hourly until acknowledged                                                         |

Combining is per class. Three outages at once is one phone call that lists all three, not three phones ringing; acknowledging that call silences exactly the outages it named, and an outage that opens afterwards rings on its own. The same holds for the hourly degraded text. Replying `1` acknowledges everything the newest text named; `ALL` acknowledges every open alert.

The hourly interval is `DEGRADED_TEXT_INTERVAL` in `server/monitoring/alerting.py`; the outage call interval is the console's **Repeat every** setting.

Prepare a Twilio account with a voice-capable number, then in the console enter the Account SID, the Auth Token, and the caller ID. The caller ID must be a number the account owns or a verified outgoing caller ID; **Test connection** checks the credentials and refuses a caller ID Twilio will not accept. Add one or more on-call numbers in E.164 form, use **Test call** and **Test text** on one of them to prove the whole path, and only then tick **Call on outages**.

Texting is a separate switch alongside the calls. **Test connection** reports whether the caller ID can send messages: a number the account owns usually can, while a merely verified caller ID can place calls but never text, and the switch stays refused until Twilio confirms otherwise. With texting on, each alert sends one message per recipient when an outage opens and one when the sensor recovers; the calls carry the repetition so the thread does not fill up. Texts also carry degraded alerts, since those are never called, so turning texting off leaves them visible only in the console.

To let someone acknowledge by replying, Twilio has to deliver that number's inbound messages here. The panel shows the reply URL and a **Point Twilio here** button that sets it through the API; you can also paste the URL into the number's messaging webhook in the Twilio console. Replying `1` acknowledges the alert at the top of the thread and says how many are left; `ALL` acknowledges every open alert. A text from a number that is not on call is ignored without a reply.

The defaults page after one minute of sensor silence, call every enabled number, repeat every sixty seconds, ring for thirty, and never stop until acknowledged. Answering and pressing **1** stops the calls for that sensor and hangs up the other phones still ringing; the alert stays visible in the console until recovery is confirmed. An operator can also acknowledge from the console. Setting **Stop after** to a non-zero number of rounds caps the calling.

The **Health details** column shows the latest independent health report and expands to list each monitored function, its diagnostic message, and any reported recovery attempts. Zero-attempt counts are omitted. The general status and degraded start time remain visible without expanding the functions. Older agents show an upgrade notice: their Running status establishes contact only, not function health. Ordinary offline alerting still applies to them.

Deploy server migration `0057_agent_degraded_since` with the server to enable persistent degradation timing and normalize the old configurable silence threshold to 60 seconds. Rebuild and upgrade sensors for accurate dependency/retry accounting and scanner stuck-worker recovery; the server can show the new general statuses for older health-capable agents during rollout.

The agent version is refreshed from authenticated health reports even when inventory collection or telemetry upload fails. Queued pre-upgrade inventory cannot overwrite that live version; legacy agents still report their version through inventory.

Agent 0.1.12 restores device discovery and per-IP host counters on WAN captures that expose public bridged VM MACs. Inventory now preserves assigned subnet prefixes so application ownership includes the connected VM subnet when capture moves from a numbered bridge to its unnumbered physical uplink. Deploy the control plane, then use **Upgrade** on each affected agent; a web-only deployment cannot change packet attribution in an already running sensor. Verify that device last-seen timestamps advance and that host and application rankings receive new traffic after the next inventory and completed minute arrive.

Agent 0.1.11 fixes two collection errors that retries cannot repair in older builds. Inventory skips non-interface sysfs class files such as `/sys/class/net/bonding_masters`, which previously caused `NotADirectory` (OS error 20) and prevented fresh bridge/uplink topology from reaching the panel. The evidence worker distinguishes the agent's `ddos-baselines-*.json` state from capture manifests; an empty baseline previously appeared as an unreadable manifest with `JSON Data at line 1, column 28`. Keep those baseline files: they contain learned DDoS state, not broken evidence. Genuine unreadable capture manifests remain retained and reported as failures.

| Alert                                  | When it opens                                                                                                                 | First checks                                                                                         |
| -------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| Agent offline                          | No authenticated contact for 60 seconds                                                                                       | Host power, agent service, outbound connectivity, panel availability                                 |
| Agent degraded: function/configuration | Reported function impairment or unapplied configuration, with continuous degradation for 15 minutes                           | Expand functions; inspect logs, capture permissions, interfaces, disk space, and dependency failures |
| Agent degraded: health reports         | Health reports missing for 60 seconds while contact continues, or required functions absent; 15-minute degraded grace applies | Reporter/service health and the direct health HTTP path                                              |
| Agent degraded: telemetry              | No receipt or processing progress for 120 seconds; 15-minute degraded grace applies                                           | Uploader/spool state, telemetry workers, and database writes                                         |

Brief interruptions are recorded before they qualify for paging. With the default ten-second capture interval, thirty seconds without accepted contact starts a **Node reporting gap**. Recovery closes the event at the server's receipt time, even if the whole gap fell between dispatcher ticks. These records and the WAN timeline show seconds, and do not send calls or texts on their own. Offline alerts use a fixed 60-second threshold; the dispatcher normally checks every ten seconds, so delivery follows the next evaluation.

Migration `0032_wanavailabilityminute` preserves individual existing probe rounds for 30 days, so a failure and recovery within the same minute no longer disappear into a successful minute. Deploy the server and migration to enable this logging; no agent update or additional node reports are required. Earlier minute-only history cannot reconstruct brief outages retroactively. Existing WAN probes run approximately every 10–16 seconds per WAN, so a shorter physical outage entirely between probes can still go unobserved. A reporting gap may also reflect upload delays or server latency, which is why it is labeled separately from a failed WAN probe.

The agent retries failed functions with backoff and reopens their resources. Repeated failures stay visible while retries continue. A stuck worker or health reporter causes the installed systemd watchdog to restart the main agent. The separate scanner service exits on an internally detected stuck worker; its existing systemd restart policy restarts just the scanner and reaps its Nmap children. A legitimate long scan keeps its supervisor polling and is not a stuck-worker timeout. This does not impose a runtime limit on Nmap or guarantee detection of a stuck child whose supervisor is still responsive. The panel does not depend on receiving a final failure callback: its own deadlines detect missing contact, missing health, and missing telemetry progress. For an upgraded agent, an alert resolves only after healthy reports span at least 30 seconds and current telemetry has been processed since that healthy period began. Existing incidents also require 30 seconds of fresh reports after the panel observes the issue clear. Any recurrence resets that confirmation. Acknowledgement silences paging; it does not mark the function healthy.

Twilio has to reach this console to deliver the keypad digit, so `L1_PUBLIC_URL` must be the public HTTPS URL operators use. The console shows the callback URL it will hand to Twilio and refuses to report itself ready when that URL is not public. A callback whose signature does not verify against the stored Auth Token is rejected, so changing the public URL without saving it here silently breaks acknowledgement — the health chip reports `Public URL required` when it does.

Failures are recorded whether or not Twilio is configured. Offline agents stay visible with their last contact, an `agent_offline` event, and unverified WAN status; sustained reachable failures use `agent_degraded`, with their specific reason in the detail. Historical issue types remain readable. Twilio controls delivery to people, not whether the failure is recorded.

The dispatcher runs inside the web container and reports a heartbeat; the health chip reads `Dispatcher not running` when no heartbeat has arrived in three minutes. Set `L1_ALERT_DISPATCHER=0` to run it elsewhere as `python manage.py run_alert_dispatcher`, which is safe to run in more than one process. Railway cron jobs cannot run more often than every five minutes, so the dispatcher is a loop rather than a scheduled job.

Monitor the public panel and dispatcher heartbeat from a separate monitoring service, and periodically verify a test call and acknowledgement. The panel cannot deliver its own alerts during a total panel/database outage, and a healthy agent cannot repair that failure remotely.

### Health rollout and failure drills <a href="#health-rollout-and-failure-drills" id="health-rollout-and-failure-drills"></a>

Deploy database migrations, the web process, telemetry worker, and dispatcher before upgrading agents. Upgrade one staging agent first using the published installer, which installs both the new binary and the matching systemd unit. Confirm that function health reports arrive, current telemetry advances, and the table settles to **Healthy**. Check the installed service configuration:

```sh
systemctl show l1-monitor-agent.service -p WatchdogUSec -p Restart -p NRestarts
journalctl -u l1-monitor-agent.service --since '10 minutes ago'
```

`WatchdogUSec` must be nonzero; copying only the binary leaves an older unit without hung-process recovery. Preserve the agent key, instance ID, and spool when upgrading. Observe the staging agent through a recovery and confirm the alert clears before upgrading the remaining devices.

Use isolated test fixtures or a disposable staging agent and test recipients for failure drills. Do not stop production monitoring or alter Proxmox bridges to simulate a failure. In a test harness, inject a worker error until recovery is exhausted, then restore successful operation: verify a function-specific failure alert and sustained recovery. Suppress health reports while continuing telemetry to verify **Health reports missing**; suppress all callbacks to verify **Agent offline**; keep health healthy while preventing fresh telemetry receipt or processing to verify **Telemetry stalled**. Advance the test clock past the deadlines rather than waiting or manipulating a production clock. Verify that omitted function reports, a single successful report, a process restart, stale report replay, and acknowledgement do not prematurely clear an active health failure.

The regression suites exercise these cases without contacting a real agent or calling on-call numbers. Run the Django suites from `server/` in the configured Python environment and the UI suites from the repository root:

```sh
python manage.py test monitoring.tests.test_agent_health monitoring.tests.test_health_alerting --settings=l1monitor.test_settings
```

```sh
node --test server/monitoring/tests/agents_ui.test.js server/monitoring/tests/alerting_ui.test.js
```

For a watchdog drill, use a disposable Linux VM with the installed unit and pause its test agent process; confirm systemd replaces it and the panel records any failure lasting past the silence threshold. Restore the VM after the test. The healing scope is the monitoring service and its resources: WAN packet loss or an unreachable probe target must never trigger host network, firewall, VM, or route changes.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.layeronecloud.com/network-monitor/operations/alerts.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.