> For the complete documentation index, see [llms.txt](https://docs.layeronecloud.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.layeronecloud.com/platform/monitoring.md). # Monitoring, outages and incidents monitoring.Monitor holds site, optional device link, type, target, cadence, timeout, thresholds, allowed worker groups, current state and claim fields. ## 3.1 Monitors, checks and worker claiming `monitoring.Monitor` holds site, optional device link, type, target, cadence, timeout, thresholds, allowed worker groups, current state and claim fields. `monitoring.MonitorResult` holds worker group, region, latency, response code, error metadata, before/after state and a state-change flag. Check types: HTTP (urllib), TCP (socket connect), TLS (socket + certificate handshake), ping (host OS `ping`). **Claiming is database-backed and lease-based.** `claim_due_monitors` uses conditional updates against unexpired claims, so if two containers ask for due checks at once only one can acquire a given monitor. Claims expire after a short lease so a crashed worker does not permanently block execution. `execute_claimed_monitor` calls `renew_monitor_claim` immediately before running the check: a batch is claimed with one lease but executes sequentially, and a single check can run for `timeout_seconds`, so later monitors in a mass-outage batch are reached after the batch lease lapsed — without the renewal, finished results were rejected and discarded during exactly the event monitoring exists for. Renewal is conditional on the claim token, so a worker whose monitor was taken over skips the check rather than double-reporting. **Worker-group eligibility is filtered in SQL, not after slicing the due window.** Otherwise the backlog of a group whose workers are down (ever-older `next_check_at`, permanently at the front of the queue) fills the window and a healthy group fetches nothing but skippable rows, silently stopping its own checks too. For monitors eligible in more than one online group, the preferred group rotates from the most recent result, so a fast local worker cannot starve every remote region out of status history. Worker groups: `group-1` Florida USA, `group-2` Frankfurt Germany, `group-3` Virginia USA. An empty allowed-group list means any configured group may claim. Execution paths: the packaged DB-backed worker (`manage.py run_due_monitors --loop`, trusted internal hosts, authenticates through `DATABASE_URL`) and the **standalone agent** over the REST API ([3.8](#id-38-monitoring-and-looking-glass-agents)). `--loop` wraps its body: a transient database failover logs, backs off (interval doubling to `MAX_BACKOFF_SECONDS`) and retries rather than ending the process, because an exit would stop that whole region's monitoring silently. A one-shot run still fails loudly so cron and operators see the error. ## 3.2 Ping semantics **A ping check is `PING_PACKET_COUNT` (5) packets, and its latency is parsed, not measured.** One packet could not tell a dead host from a dropped datagram, so ordinary loss published outages. * Every ping implementation exits 0 when **any** packet returns, which is the fix and also the next trap: 4-of-5 lost exits 0 too, and for a network provider that link is unusable. So the loss percent is parsed and anything above `PING_DEGRADED_LOSS_PERCENT` (50%) is `DEGRADED`. * **Latency comes from ping's own `min/avg/max` line.** Packets are spaced a second apart, so wall clock around five of them is \~4000 ms for a host answering in 0.5 ms — recording that would flatten every latency chart. When the summary cannot be parsed the result carries no latency, which is an honest gap rather than a fabricated number. * **The subprocess budget is `timeout_seconds + PING_PACKET_COUNT + 2.`** A silent target holds the command for the gaps between packets *plus* the wait for the last reply; the old `timeout_seconds + 2` killed every real outage mid-run, turning it into a `TimeoutExpired` with no loss figure and no output. * **Do not add flags to `_ping_command`.** An inter-packet interval (`-i 0.2`) would be four times quicker, but a build that rejects the flag exits non-zero, which reads here as an unreachable target — and every ping monitor in a region turning into a published outage is far worse than a check that takes four seconds. The remote agent runs on hosts nobody here controls. * `LayerOne Agents/layerone_agent.py` keeps its own copy for remote workers and `AgentPingParityTests` holds the two together: drift means the same target is reported differently depending on which region checked it, which looks exactly like a regional outage. ## 3.3 Inconclusive results **A check that could not be run is `Inconclusive`, not `Failing`.** A ping that gets no reply is downtime; a ping that could not be *sent* says nothing about the target. Inconclusive results change no monitor state, touch neither counter, open no incident and no outage, and are subtracted from the measured window rather than counted as downtime. | Condition | Classification | | ------------------------------------------- | ----------------------------- | | `ping` exits non-zero (sent, no reply) | Failing | | `ping` cannot resolve the host | **Inconclusive** | | `OSError`/`SubprocessError` spawning `ping` | **Inconclusive** | | `subprocess.TimeoutExpired` on `ping` | Failing (target unresponsive) | | `socket.gaierror` on HTTP/TCP/TLS | **Inconclusive** | | Connection refused, reset, timeout | Failing | | Unexpected prober exception | Failing | | Unsupported monitor type | **Inconclusive** | **An unexpected exception stays `FAILING` on purpose.** The common case is a crashed service answering the port with a non-HTTP banner (`http.client.BadStatusLine`), which is the target being broken. Calling that inconclusive would stop detecting real outages, which is worse than the reverse. Keep the scope narrow: only positively identified prober faults. Name resolution is a prober fault because a broken resolver in one container makes every monitor fail to resolve at once, and outages auto-publish. A genuine total DNS failure still surfaces: the monitor stops producing conclusive results and coverage staleness flips it to Unknown. **Coverage staleness measures `Monitor.last_conclusive_check_at`, never `last_checked_at`** — a prober that keeps running but cannot reach anything advances the latter forever while observing nothing. `monitor_coverage_is_stale` treats a monitor as unobserved once that timestamp is older than `interval_seconds × STALE_COVERAGE_INTERVAL_MULTIPLIER`, with a `STALE_COVERAGE_MINIMUM_SECONDS` floor, and the `monitoring.flag_stale_monitors` beat task writes the same judgement back to `Monitor.current_status` so the console agrees with the public page. Sustained inconclusive results escalate through `mark_stale_monitors_unknown`, which opens a **private** monitoring-health incident: losing sight of a target is an operations problem, never a customer notice. ## 3.4 Reporting a failure: two gates, and a confirmed outage backdates past both `failure_threshold` counts checks and `confirmation_seconds` (default **180**) measures wall clock. A monitor only leaves Operational once **both** are satisfied, and the slower one wins. `0` restores check-count-only behaviour. The same check count means one minute of patience at a 30 s interval and ten at a 5 m one; what an operator wants to say is "do not tell anyone until it has been down for three minutes". **"Continuous" is the operative word.** `Monitor.failing_since` holds the first failing check of the run and is cleared by any **passing** check, so three minutes means three unbroken minutes. An **inconclusive** result does not clear it: a prober that stops working mid-outage is not the target recovering, and resetting there would let a flaky worker hold a genuinely dead target below the reporting threshold forever. **A confirmed outage backdates `started_at` to `failing_since`, not to the confirming check.** Writing no row until confirmation is what keeps blips off the SLA; timing from confirmation *on top of that* would subtract three minutes from every outage and publish a ten-minute failure as seven — getting the uptime figure wrong in the platform's own favour, the one direction a status page must never be wrong in. Those are two separate decisions and only the second one moved. The cost is accepted deliberately: a genuine outage is published three minutes later than it was, because the status page exists to tell customers about real outages and it was telling them about packet loss. ## 3.5 Outages and time-weighted uptime **Uptime is time-weighted, computed from `MonitorOutage` intervals, never from a check ratio.** `passing / total` is a function of check cadence rather than of time: one failed sample in 2,880 reads as 99.97% whether the target was gone for thirty seconds or an hour, and it cannot answer the only question an SLA asks. `MonitorOutage` stores `started_at`, `ended_at`, `duration_seconds`, `kind`, `confirmed_regions`, `counts_against_sla`, plus links to the triggering and resolving results and to the incident. A partial unique constraint allows one open outage per monitor. Intervals key off the *monitor* status, not individual results, so a failing check below the threshold is a blip and no row is written. Severity only rises within one interval, so a service sliding Degraded → Down is one outage that got worse. `pause_monitor` closes any open interval, or a monitor paused during an incident and resumed a week later would accrue a week of unmeasured downtime. `apps/monitoring/uptime.py` owns the math (`services.py` re-exports it lazily; **the import has to stay lazy or it is circular**): ``` uptime = (measured_seconds - down_seconds) / measured_seconds ``` Two properties are load-bearing: * **Unmeasured time is removed from the denominator** rather than counted as healthy. A window the prober could not run in is not evidence of good service. * **A recorded outage counts as coverage.** The only way an interval exists is that checks ran and failed, so `coverage_gaps_in_window` subtracts outage intervals from candidate gaps; without that subtraction the measured downtime gets clamped away as unmeasured and a real outage rounds to nothing. Degraded time is reported separately rather than folded into downtime. **Percentages carry three decimals** because two saturates at 100% for real availability: 20 seconds down in 30 days is 99.99923%, which rounds to "100%" and tells the reader the opposite of the truth. Summaries expose `down_display`, `outage_count`, `measured_display`, `has_coverage_gap` and `coverage_percentage` so the UI can lead with a duration and say what the figure is based on. `MonitorDailyRollup` stores per-day down/degraded/unmeasured seconds with check counts and latency percentiles, so 90-day windows stay cheap and history survives the 90-day raw purge (hourly `build-monitor-daily-rollups`). Daily history colour comes from **measured outages**, not incidents: incidents are the operator's narrative and can be edited by hand, so deriving the strip from them let the picture drift from what was observed. `manage.py backfill_monitor_outages` replays stored `MonitorResult` rows into outages and rollups, re-applying the thresholds while walking so replayed intervals start where live code would. Idempotent, scopeable to one monitor, `--dry-run` available. Deliberately a command rather than a data migration. Outages are opened and closed inside the existing `record_monitor_result` transaction under the same `select_for_update` monitor lock that serializes incident creation, with the partial unique constraint as the database backstop. ## 3.6 Auto-published outages, and the public wording boundary **Monitor incidents auto-publish, so their public text is a security boundary.** Publication is gated on the monitor feeding a **visible** component on an **enabled** status page — an internal check must not post to the customer page, both because it would confuse customers and because it would disclose that the check exists. The update stream splits. The **private** stream carries monitor names, targets, worker regions, response codes, raw errors, command lines, stdout, stderr and tracebacks. The **public** stream is generated from the public component name and the monitor state alone. `_public_incident_title`, `_public_incident_summary` and `_record_public_incident_update` are the only functions that produce customer-visible text, and `apps/monitoring/tests/test_outage_publication.py` asserts that no target, hostname, IP, device name, command, stderr or traceback survives into any public surface — on the rendered page as well as the incident. Recovery posts a public resolution too, or a resolved outage reads as open forever. **Never widen the public side to include result detail.** ## 3.7 Incidents `incidents.Incident` (site, severity, status, source, assignee, visibility, lifecycle timestamps) and `incidents.IncidentUpdate` (internal/public per update). Numbers are assigned after insert from the database primary key (`INC-000001`), so there are no process-local counters. State updates use `select_for_update()` inside a transaction. **Incidents default to private, and so do updates.** An incident must be marked public before it appears on the public status page, and public pages render only updates individually marked public, so responders keep internal timeline notes separate from client-safe ones. Audit metadata stores update message **length**, not the body. Monitor-created incidents (`source=Monitoring`) are created by `record_monitor_result` in the same transaction as the state change, so concurrent workers cannot open duplicates. Repeat failures do not create a second active incident for the same monitor; recovery resolves it. Console review prioritises active severity and age; publication controls and the customer-facing update timeline stay distinct from internal metadata. Super-admin responders can edit `started_at`, `detected_at`, `acknowledged_at` and `resolved_at` from the console (Eastern Time). Those times are what public status shows; `created_at` stays the insert clock. Reopening an incident still clears `resolved_at`. Support remains read-only. ## 3.8 Monitoring and Looking Glass agents Remote agents are individually named, individually revocable, and installable with **one credential** instead of a shared PSK plus hand-maintained regional settings. Managed under **Settings → Monitoring agents** (create, edit, enable/disable, rotate key). * Creation returns a one-time `l1agent_...` key. The database keeps only a **SHA-256 digest** and a public key id; raw keys never appear in later pages, logs, audit events or service-instance metadata. * **The authenticated database record is authoritative for name, worker group and region.** On check-in the web app returns them. An agent cannot choose another worker group in its JSON payload, so a compromised key cannot claim another region's monitors or complete another agent's Looking Glass work. * Disabled or rotated keys fail authentication without creating service state. * Support staff are default-denied from agent credentials and configuration. Runtime protocol (HTTP Bearer): ``` POST /api/v1/agents/check-in/ POST /api/v1/agents/monitors/claim/ POST /api/v1/agents/monitors//result/ POST /api/v1/agents/looking-glass/claim/ POST /api/v1/agents/looking-glass//progress/ POST /api/v1/agents/looking-glass//result/ ``` The standalone agent runs on a small Ubuntu host with **no** Django, database credentials, Redis, Celery, SMTP or production `SECRET_KEY`. Its installer installs Python, ping, traceroute, MTR and CA certificates; creates the restricted `layerone-agent` user; installs to `/opt/layerone-agent`; stores the key in `/etc/layerone-agent/agent.env`; and enables `layerone-agent.service`. The control-plane URL defaults to `https://layeronecloud.com`; `LAYERONE_API_BASE_URL` remains an optional override, and the production installer writes only `LAYERONE_AGENT_API_KEY`. To change an agent's name, group or enabled state, edit it in the console — the process picks it up at its next check-in. To replace a credential, rotate in the console and rerun the installer. `AGENT_API_PSK` is accepted **only** as a temporary compatibility fallback for a rolling upgrade and should be removed afterwards. `AGENT_ONLINE_SECONDS` (300) decides when an agent reads as down. **Leave `STATUS_MONITOR_WORKER_GROUP` blank on web, beat and ordinary worker containers.** The web server is the control plane; it should not run monitoring checks or Looking Glass commands. ## 3.9 Infrastructure inventory `infrastructure.Device` (site, role, status, management hostname, location, notes, creator), `DeviceIPAddress` (with primary-IP support), `DeviceConnection` (router/switch/management/trunk/LAN/dependency links) and `DeviceTag`. VMs are represented as host links in the generated topology map. Inventory is **admin-only**: device names, management hostnames, locations, IPs and internal notes are never exposed to clients or public status pages, and audit events record note **length** rather than the note. The console defaults to a searchable device directory with status, role and location filters; topology is a separate view retaining the complete site's graph, so list pagination does not silently remove graph edges. Device relationships paginate independently. Operator-assigned device status is kept distinct from saved monitor observations. Known modelling debt: `infrastructure.Device` and `billing.ProxmoxNode` are two rows for the same hypervisor. Giving one an FK to the other is a data-model change nobody has taken yet. *** --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.layeronecloud.com/platform/monitoring.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.