> For the complete documentation index, see [llms.txt](https://docs.layeronecloud.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.layeronecloud.com/network-monitor/operations/proxmox-sensor.md). # Proxmox sensor The production web image builds the Linux agent and publishes only these allowlisted, credential-free artifacts without authentication: The production web image builds the Linux agent and publishes only these allowlisted, credential-free artifacts without authentication: ``` /downloads/agent/install.sh /downloads/agent/l1-monitor-agent /downloads/agent/l1-monitor-agent.sha256 /downloads/agent/l1-monitor-agent.service /downloads/agent/uninstall.sh ``` Create a site or agent in the web interface and copy its generated one-line install command into a root shell on the Proxmox host. The command downloads the installer from that Network Monitor and supplies the one-time key with `--api-key`. The installer then downloads the agent, checksum, and systemd unit from the same server, verifies the SHA-256, validates the server connection, stores the key at `/etc/l1-monitor/agent.key`, and starts the service. The generated command contains a live credential. Use it only in a trusted root shell, do not paste it into tickets or chat, and remove it from shell history after installation. The published agent artifacts never contain the key or any control-plane credentials. The image build rejects private-key material in the first-party agent sources and L1 key values in the final binary. Use **Reinstall** on the Agents page to rotate that host's key and reopen the same install-command dialog. Each installation receives a persistent identity in `/etc/l1-monitor/instance-id`. If two different installation identities use one agent enrollment within 90 seconds, the server marks the agent failed and rejects both installations. Stop or remove the duplicate, then use **Re-enable** to let the first returning installation claim the enrollment, or use **Reinstall** to rotate the key and install a fresh copy. The lockout never clears itself. ```sh systemctl status l1-monitor-agent.service journalctl -u l1-monitor-agent.service --since today /usr/local/sbin/l1-monitor-agent check --config /etc/l1-monitor/agent.toml ``` Starting with agent 0.1.16, detailed traffic history and current usage have separate upload paths. Each completed minute keeps its original minute-start timestamp and is retained in SQLite only until the server accepts it. Current readings normally upload every ten seconds as of agent 0.1.17 in one request covering all captures; they bypass the history worker and its database lock. An unsent current reading is replaced by a newer one and expires after thirty seconds. The separate health callback continues every fifteen seconds. WAN probe rounds and DDoS evidence still retain their durable delivery because they contribute to outage and incident history. The ten-second default sends one fifth as often as the earlier two-second default. Minute traffic totals still close every minute, and graphs divide each live reading by its actual measured duration. The current readouts allow thirty seconds for the next reading, so ordinary delivery delay does not make ten-second updates flicker stale. Server migration `0038` updates existing default intervals and their configuration revisions; custom intervals remain. Deploy the updated control plane first, then update the sensors. The config API advertises `live_upload_version: 1`; an agent connected to an older server keeps the legacy spooled live path. Existing queued history remains available for backfill after the update; clearing backlog is not required. When the upload queue is more than 15 seconds behind, the agent logs that historical backfill is delayed and prioritizes the newest durable WAN status. The pending count and oldest age can therefore remain elevated while the dashboard stays current; they should fall as upload capacity becomes available. Repeated `telemetry upload unavailable` messages still indicate a transport or server problem and should be investigated even if current samples occasionally get through. Reading the uploader's log lines: * A batch leaves the sensor's spool the moment the server answers 202: the server stored it in its inbox before answering, so there is nothing left to wait for. The spool therefore only grows while the server is unreachable, and the fifteen-second health report carries its size and the age of its oldest batch, which the Agents page shows as "Waiting to upload" beside the server's own "Waiting to process". Older agents logged `server accepted sequence N but has only committed M` and kept every batch until the server's commit point moved; that protocol is gone, and with it the fault about the server not committing for two minutes. * `current live upload unavailable` concerns only the current readings path. Requests time out after fifteen seconds, then wait ten seconds before the next attempt uses the latest available readings. The server batches live publication with bounded dependency deadlines and returns 503 if neither history buffering nor WebSocket delivery succeeds. Repeated timeouts still require checking network/server latency; they do not alone prove uploads are too frequent. Detailed minute history continues through its own uploader. * `telemetry upload unavailable: operation timed out` means the control plane did not answer within the upload timeout (45 seconds). The server usually still finishes applying that batch, so the retry is wasted work; a run of these on an otherwise reachable server is the control plane falling behind, not the provider rate-limiting the sensor, which would answer quickly with an error status instead. Check **Agents & Settings > Telemetry processing**: the mode should be queued and the workers reported as running. * `telemetry upload unavailable: telemetry server returned 503 Service Unavailable` is the control plane's edge answering while the application behind it did not: a deploy or restart in progress, or the application refusing uploads because its database is unreachable. Either way only the server can recover. The sensor keeps every batch, retries with a backoff from one to thirty seconds, and reports the wait on its uploader function as a note ("waiting on the server for 40s with 120 batches kept on the sensor"). It is dependency degradation, not a failed recovery attempt. A reachable agent waits 15 continuous degraded minutes before alerting; loss of contact alerts after one minute. Only the sensor's own storage failing, such as a spool that cannot be read or written, still counts as a fault it has to heal. * `telemetry backfill is behind: N batches queued, oldest S old; ... draining at R batches/s, about T to clear` is logged every fifteen seconds while behind. It gives the size of the debt, how old the oldest queued reading is, and whether the queue shrank since the last line; `not shrinking` means uploads are not outpacing new readings. While behind, up to eight old batches are uploaded per cycle between refreshes of durable WAN readings and legacy spooled live frames, and that group is sent at once rather than one request after another: one upload at a time would drain at one batch per round trip however fast the control plane commits. Acceptance does not depend on order, so the group is settled in sequence order once the answers are in. Current live and WAN readings stay sequential, being one item each and wanting low latency rather than throughput. `UPLOAD_LANES` in `agent/src/main.rs` is the group size. Legacy live frames older than two hours are skipped as exact gaps (`skipped N live frames older than two hours`): the console cannot show them and every minute rollup, event, probe round and inventory is kept. The server records the skip as an informational event, not a discarded-data warning. * `agent health callback unavailable` is the separate 15-second health report failing for the same reason; the next report carries fresh state. * `current live upload unavailable: ...; next attempt uses fresh readings` is the disposable two-second live path failing once. The next reading replaces it, the sensor records the reason on its live function without declaring it broken, and nothing is paged. A run of these is worth reading: an interval the server calls invalid usually means the node's clock is minutes ahead of the server's. * `server rejected FIFO minute sequence N: ... 400 Bad Request` means the control plane refused that minute outright, and the sensor replaced it with a permanent gap. Application facts used to cause this whenever their strict internal accounting failed a check. The server now judges each application fact on its own: a malformed row is skipped, every other row is stored, and coverage summaries that disagree with the rows are recorded rather than fatal. Both cases open one `application_usage_discrepancy` warning event per sensor carrying the reasons and counts; only a counter past the database bound still skips a whole minute's facts, under `application_usage_rejected`. A 400 on a minute is therefore now a genuinely malformed upload. * The Applications card saying *Traffic seen, but none of it could be placed* means the sensor saw traffic but could not tell which side was yours, so nothing was attributed. Address rules cover private ranges, the capture's connected prefixes and addresses learnt from ARP. On an uplink capture two more decide it: the router's MAC, read from the host's default route and neighbour table, marks the Internet side of every frame, and the capture's DDoS-protected prefixes count as local. Guests with public addresses are therefore attributed to their own MAC without any operator setting; if the card still shows this, check that the WAN capture sits on the uplink and that its protected prefixes cover the site's address space. ## Discard all local agent backlog The agent's `clear-backlog` command discards buffered telemetry, including protected DDoS destination batches, the separate destination databases, and generated local packet-capture files and manifests. This is permanent: the discarded history cannot be recovered. It preserves the enrollment key, instance identity, desired configuration, detector baselines, and unrelated files. Stop the service first so capture workers and uploaders cannot refill the queues. Use the service account to preserve state-file ownership. The command works offline and reads the enrolled agent ID from `desired-config.json`. ```sh sudo systemctl stop l1-monitor-agent.service # Preview counts and bytes; no backlog is removed without --yes. sudo -u l1-monitor /usr/local/sbin/l1-monitor-agent clear-backlog # Discard the local backlog. sudo -u l1-monitor /usr/local/sbin/l1-monitor-agent clear-backlog --yes # Restart only after the clear succeeds. sudo systemctl start l1-monitor-agent.service ``` Pass `--config /path/to/agent.toml` for a non-default installation. Install the updated binary before using this command; old binaries do not recognize it. Updated agents and maintenance commands share a process lock and refuse to run against the same state directory concurrently. Always stop older running agents explicitly as they predate that lock. The telemetry clear is transactional and leaves only small, exact sequence-gap notices for the server to acknowledge. The server records the intentional loss with reason `operator_cleared_backlog`; new collection continues after restart. Data already accepted by the server, including its Redis queue, remains there. Freed SQLite pages are reusable; clearing rows does not necessarily shrink the spool file on disk. Evidence files are removed from disk. If evidence cleanup fails after telemetry was cleared, the error reports partial progress; leave the agent stopped and retry after addressing the reported error. Symlinks and unexpected file types at deletion targets are refused. ## Upgrade the agent Use **Upgrade** on the Agents page to generate and copy the host-specific upgrade command. The existing key remains in place. The equivalent manual command is: ```sh ( set -eu l1_installer=$(mktemp /tmp/l1-monitor-install.XXXXXX) trap 'rm -f -- "$l1_installer"' EXIT l1_server='https://your-service.up.railway.app' curl --proto '=https' --tlsv1.2 --fail --silent --show-error --location \ "$l1_server/downloads/agent/install.sh" --output "$l1_installer" /bin/sh "$l1_installer" update --server-url "$l1_server" ) ``` Each run uses a unique temporary file and removes it on success or failure. This avoids write failures such as `curl: (23)` caused by a leftover fixed installer filename in `/tmp`. Update from a local binary instead: ```sh ./install.sh update --binary /root/l1-monitor-agent ``` An automated download must use HTTPS and include the release SHA-256: ```sh ./install.sh update \ --download-url https://downloads.example.com/l1-monitor-agent-amd64 \ --sha256 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef ``` When a key is rotated, paste the new key into an update command (passing the installed binary is safe), then clear that command from shell history: ```sh ./install.sh update \ --api-key 'l1a_REPLACE_WITH_THE_NEW_KEY' \ --binary /usr/local/sbin/l1-monitor-agent ``` Uninstall while preserving the outage spool for recovery, or explicitly purge it as well: ```sh ./uninstall.sh ./uninstall.sh --purge ``` WAN membership is automatic and identical across all features. Assigned public IPs identify WAN owners; bridge ports determine the physical, bond, or VLAN uplink to capture. For example, `vmbr0` owning `38.75.229.114` with bridge port `nic2` creates a WAN capture on `nic2`, retaining `vmbr0` as the IP owner and probe-source owner. If the port is `bond0`, capture uses the bond once, never its member NICs. Other non-overlapping infrastructure bridges remain LAN. Private/CGNAT-only interfaces and virtual-only SDNs such as `n0000002` are not WANs. Per-VM `fwbr`, `tap`, `veth`, and related firewall plumbing is ignored. The configuration screen shows the resolved interfaces and public IP owners read-only. Operators may change DDoS sensitivity, thresholds, evidence, and additional protected CIDRs, but cannot add a WAN or override its source IP. An incomplete topology refresh keeps the last confirmed mapping and displays a stale warning; an unresolved public interface displays a partial warning. Link-down does not remove membership while its public IP remains assigned. IPv6-only WANs retain traffic/DDoS monitoring and show IPv4 probing unavailable. Roll out the control plane and migration `0029_shared_wan_resolution` before agent **0.1.9**. The migration retires manual WAN choices and preserves their DDoS preferences and historical telemetry. Old sensors show upgrade-required until a complete versioned topology inventory arrives; manual WAN flags are not a fallback. Verify `vmbr0 -> nic2` (or the bond) on the agent configuration, WAN sidebar, and DDoS page after upgrading. Compare its traffic with Proxmox and switch counters. Firewall-rule automation is not enabled by this change. Public-IP removal or topology changes retire current WAN membership while retaining historical probe, usage, and DDoS records. Configuration is polled every 30 seconds; inventory, probe, and dashboard refresh intervals determine when a changed binding appears everywhere. Capture workers retry automatically after libpcap failures. When capture interfaces change, the agent discards obsolete queued telemetry using an explicit sequence-gap record so new utilization and device observations are not blocked behind data generated for the old capture set. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.layeronecloud.com/network-monitor/operations/proxmox-sensor.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.