> For the complete documentation index, see [llms.txt](https://docs.layeronecloud.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.layeronecloud.com/network-monitor/reviews/reliability-2026-09-05.md).

# Reliability review — 2026-09-05

Scope: the Rust sensor, durable telemetry delivery, metering, WAN uptime, alerting, frontend freshness, authentication, integration inputs, and deployment configuration.

Scope: the Rust sensor, durable telemetry delivery, metering, WAN uptime, alerting, frontend freshness, authentication, integration inputs, and deployment configuration. Existing worktree edits were preserved. Nothing was deployed; no production migrations, calls, texts, or external account changes were made.

## Implemented fixes <a href="#implemented-fixes" id="implemented-fixes"></a>

* Upload authentication and deployment failures retain queued usage. Payload rejection and spool retirement use explicit sequence gaps, and acknowledgements must identify the exact submitted sequence.
* Capture reconfiguration retains minute, WAN, event, and DDoS history. Graceful shutdown drains workers and saves partial minutes. Revision-only changes no longer restart captures unnecessarily.
* Idle libpcap captures remain responsive; invalid sysfs reads cannot reset byte baselines to zero and later bill lifetime counters. Discovery caches are bounded.
* ICMP replies must belong to the target and local address being probed, and receive waits respect the remaining deadline.
* Live samples with different measurement spans remain separate, preserving rates.
* Distinct partial-minute envelopes merge their counters; retransmission of an already committed envelope remains idempotent. Delayed device traffic invalidates cached hourly totals; readers share the rollup lock for a consistent result.
* Retired capture usage can use recorded WAN membership when processing backfill. Future rows do not enter current device totals; monthly WAN usage uses UTC.
* Queue processing uses the Redis acceptance timestamp for sensor contact. Accepted HTTP retries still refresh contact; backfill cannot impersonate a new heartbeat or rewind a newer contact timestamp.
* WAN health requires fresh probe state and an answering target, in addition to a reporting, enabled sensor and carrier. Uptime and outage timelines count the same eligible minute buckets.
* Inventory timestamps prevent older snapshots, including snapshots with no interfaces, from restoring retired topology or downgrading version/hostname. Malformed identities, unrepresentable metrics, future observations, and incomplete binding events receive controlled payload errors.
* Delayed IP claims cannot create a false conflict with a much newer owner or rewind an active conflict. Conflict cleanup does not overwrite a newer update.
* SMS recipient fanout is durable before provider work begins. Known-safe failures retry; ambiguous POST outcomes stay visible without automatically duplicating a billable message. Recovery goes only to the original notified numbers.
* Duplicate inbound SMS replies cannot acknowledge another incident. `ALL` handles every eligible incident. Callback and REST completion updates preserve newer lifecycle states. Call rounds stop progressing after acknowledgement or recovery.
* Provider failures remain visible in dispatcher health. Invalid Unicode webhook signatures and invalid timing values are rejected safely.
* WAN cards stop counting healthy uptime when polling fails or expires. Operational tables refresh, stale responses cannot replace newer state, and overview loading retries and refreshes capture configuration after reconnecting.
* OTP input rejects Unicode digits safely. Password preauthentication expires and becomes invalid after password changes. Invalid site selectors and corrupt release markers have controlled behavior.
* PostgreSQL URLs preserve encoded credentials and connection TLS options. Malformed Pulsar inventory fails without partial writes. Nginx sets explicit proxy headers and rate-limits the second-factor routes.

These changes add migrations `0027_alertinboundmessage` and `0028_agent_inventory_observed_at`. Server changes must be deployed before the updated agent because partial-minute preservation requires additive ingestion.

## Verification and current blocker <a href="#verification-and-current-blocker" id="verification-and-current-blocker"></a>

* Initial server baseline: 474 tests passed.
* Agent review: 117 Rust tests and strict Clippy passed in an isolated copy using the original protocol schema, at `/private/tmp/l1-agent-review.28Pv7C`.
* Frontend review: 118 JavaScript tests passed before the overlapping work began.
* Security/settings/release/onboarding: 41 targeted Django tests passed; Pulsar integration: 74 targeted tests passed.
* Alerting: 87 tests passed with normal migrations before the overlap. Following the final regressions, 89 of 90 passed against disposable syncdb; the remaining retention test needs the unmanaged metric table created by normal migrations and had passed in the earlier run.
* Added server reliability regressions in `test_reliability.py`, with related ingestion, conflict, worker, and WAN tests. Final combined validation is pending.

Another task began changing the same workspace during final verification. Its WAN topology protocol fields initially lacked corresponding migrations and Rust initializers. A subsequent migration appeared, but later test startup failed on an in-progress import of `monitoring.agent_health` from alerting. These overlapping changes were preserved and are not part of the completed review's validation. The current combined tree must not be described as passing or ready to deploy.

After the overlapping work is complete, run:

```sh
.venv/bin/python server/manage.py makemigrations --check --dry-run --settings=l1monitor.test_settings
.venv/bin/python server/manage.py test monitoring --settings=l1monitor.test_settings --noinput
node --test server/monitoring/tests/*.test.js
cargo test --locked --offline --manifest-path agent/Cargo.toml
cargo clippy --locked --offline --manifest-path agent/Cargo.toml --all-targets -- -D warnings
git diff --check
```

## Deployment verification still needed <a href="#deployment-verification-still-needed" id="deployment-verification-still-needed"></a>

PostgreSQL/Redis concurrency, nginx runtime configuration, real Twilio delivery, and Linux/Proxmox capture and signal shutdown were not exercised locally; the required services/platform were unavailable. Multi-WAN probe egress depends on host source-policy routing. Distinct live windows can increase request volume on hosts with many captures and need a representative load test. Abrupt power or kernel failures can still lose observations collected before durable spooling.

The existing paging feature covers offline sensors. WAN status monitoring does not by itself establish separate WAN or per-VM paging, and passive traffic is not proof that an individual VM/service is reachable. Those alert targets remain a user scope decision. An outage of the control plane itself requires an independent external monitor capable of alerting while this service is unavailable.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.layeronecloud.com/network-monitor/reviews/reliability-2026-09-05.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.