> For the complete documentation index, see [llms.txt](https://docs.layeronecloud.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.layeronecloud.com/network-monitor/operations/telemetry.md).

# Telemetry processing and WAN status

committed. Queue and commit with workers (the default) has the upload request store the batch in the PostgreSQL inbox and answer 202 at once, which is when the sensor deletes its…

## Telemetry processing <a href="#telemetry-processing" id="telemetry-processing"></a>

**Agents & Settings > Telemetry processing** decides how sensor uploads are committed. *Queue and commit with workers* (the default) has the upload request store the batch in the PostgreSQL inbox and answer 202 at once, which is when the sensor deletes its copy; the worker supervisor in the web container keeps the configured number of `process_telemetry` workers alive. Each worker claims one sensor at a time with a skip-locked lock, applies that sensor's rows in sequence order in one transaction and deletes them, so sensors are committed in parallel while each sensor's history stays ordered, and the count can change at any time without handing anything over. The section shows how many workers are alive, when they last reported, how many batches are waiting and how old the oldest is, so a backlog is visible before sensors start logging about it. Raise the worker count when the waiting count keeps growing, up to 20. The number applies to each web replica, so the fleet runs that many times the replica count.

Each worker holds one database connection for its whole life. `CONN_MAX_AGE` is zero, which is right for the web service where a connection belongs to a request, but a worker is a loop: with that setting it opened a new PostgreSQL backend on every poll, and a fleet polling a few times a second becomes thousands of connections a second. That starves the web service first, so the symptom is upload, config and live-reading timeouts while the queue stops draining. Count connections before blaming the workers:

```sh
psql "$DATABASE_URL" -c "SHOW max_connections;"
psql "$DATABASE_URL" -c "SELECT count(*) FROM pg_stat_activity;"
```

The worker count multiplies by the replica count, so 10 workers across 42 replicas is 420 connections plus the web service's own. Since only one worker commits a given sensor at a time, 1 or 2 per replica costs nothing in throughput and leaves the database room to answer everything else.

A worker claims a sensor with a PostgreSQL advisory lock, deliberately not by locking the sensor's row. Every upload locks that row to record contact, so a run that held it made a sensor's own uploads wait for its backlog to drain: the drain throttled its own supply, the queue read empty however many workers were running, and the sensor's spool never emptied. With the claim off that row, a run can be long (1,000 rows or ten seconds) without holding up collection.

**Only one worker commits a given sensor at a time.** A sensor's batches are applied in sequence order, so its queue is held by one worker until that run finishes; the rest move on to other sensors. *Sensors queued* in the same panel is therefore the ceiling on how many workers can be busy, whatever number is running: with one sensor uploading, one worker commits and the others idle, and that is correct rather than a fault. Workers start their scan at different points in the queue so a fleet lands on as many sensors as it has, and a sensor with a day of backlog never hides one whose batch arrived a second ago. Making a single sensor faster is a matter of the run size and the work per batch, not of adding workers.

Raising it past the number of sensors that have a backlog does not help: a sensor's batches are committed in sequence order, so one worker commits one sensor at a time, and every extra worker is a process holding a database connection. If the host cannot start the number asked for, the section reads "N of M worker(s) running; this host would not start more". *Apply inside the upload request* commits each batch before answering the sensor and suits only a very small site. The same section sets how long the rolling-day application ranking is shared between tabs. The Agents page shows each sensor's own two numbers: batches still waiting to upload (from its health report) and batches the server holds but has not applied yet.

## WAN status states <a href="#wan-status-states" id="wan-status-states"></a>

A WAN card has three states, not two. *Up* and *Down* are verdicts on the link from probe rounds that arrived. *Unverified* is a fact about the reporting path: the sensor is silent, or it is reporting but no probe result has reached the control plane within the ninety-second grace. Unverified time is shown on its own ("Unverified · 24h") and is never booked as downtime, and the 24-hour uptime is judged over measured minutes only. That way an outage of the telemetry path cannot read as an outage of the ISP, and a stopped sensor cannot read as a perfect link either: the card says "Unverified", the timeline draws the stretch grey, and continuous uptime shows a dash until measurement resumes.

Since agent 0.1.21 the newest probe round also rides the fifteen-second health report, which keeps flowing when bulk telemetry does not. The control plane applies it to the link's current state only (link up, targets reachable, last seen); minute history and precise samples still come from the durable round, so nothing is counted twice. During an ingestion outage the WAN card therefore stays current while the history behind it fills in once uploads resume.

## When the server refuses uploads <a href="#when-the-server-refuses-uploads" id="when-the-server-refuses-uploads"></a>

The first thing to check is the reason the console now prints. A schema that has fallen behind the models looks exactly like a full disk from the outside: reads answer, writes fail. `python manage.py verify_schema` compares the tables the upload and alert paths write against the models and names any missing column; `start.sh` runs it after every `migrate` so a deployment log carries the answer. If `migrate` reports nothing to do while `verify_schema` reports drift, a migration was edited after it was applied and needs a repair migration, as `0042_repair_inbox_columns` does for the telemetry inbox.

`/agent/v1/telemetry` answers 503 `telemetry storage is unavailable` when the database will not store an upload. Sensors keep every batch and backfill once it clears, so nothing is lost, but nothing arrives either. A database that can be read but not written is the dangerous case: every page renders, health reports land, the queue reads empty, and only the sensors know. It is almost always a full PostgreSQL volume.

Since this build the instance stops pretending. Each upload records its outcome in the web process, `/healthz/` fails with `telemetry_storage_failing` once refusals have persisted for a minute, and **Agents & Settings > Telemetry processing** reads *Uploads refused* with the database's own reason. On the Agents page the sensor's own row shows the other half: "Waiting to upload" grows while "Waiting to process" stays clear. That combination — a growing sensor backlog with an empty server queue — means acceptance, not processing.

To confirm and clear a full volume:

```sh
psql "$DATABASE_URL" -c "SELECT pg_size_pretty(pg_database_size(current_database()));"
psql "$DATABASE_URL" -c "SELECT relname, pg_size_pretty(pg_total_relation_size(relid)) FROM pg_catalog.pg_statio_user_tables ORDER BY pg_total_relation_size(relid) DESC LIMIT 15;"
psql "$DATABASE_URL" -c "INSERT INTO monitoring_telemetryinbox (agent_id, boot_id, sequence, kind, body, received_at, attempts, last_error) VALUES (NULL, '', 0, '', ''::bytea, now(), 0, '') RETURNING id;"
```

The last statement is expected to fail on its foreign key; a failure naming disk space instead identifies the cause. Raise the volume, then run `python manage.py maintain_metrics` to apply retention. Daily fact partitions are the usual growth: `monitoring_application_usage_minute` and the endpoint tables. Sensors resume on their own, oldest first, and the Agents page backlog should fall steadily.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.layeronecloud.com/network-monitor/operations/telemetry.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.