> For the complete documentation index, see [llms.txt](https://docs.layeronecloud.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.layeronecloud.com/network-monitor/performance.md).

# Proxmox performance release gate

Source project: `L1 Network Monitor` (migrated from `tests/performance/README.md`). Run commands from the code checkout and directory specified below, not from this documentation repository.

Run this gate on the intended Proxmox VE 9 hardware, not in CI.

1. Connect two traffic-generator ports through one selected test bridge.
2. Record generator byte/packet totals and Linux interface counters.
3. Replay a representative mix at 1, 5, and 10 Gbit/s for 15 minutes each.
4. Include small-packet, MTU-sized, TCP, UDP, VLAN, IPv4, and IPv6 phases.
5. Export interface, device, `pcap_received`, `pcap_dropped`, `parser_dropped`, and coverage data.
6. Require interface totals within ±1% of the Linux/generator reference.
7. When libpcap and parser counters both report zero drops, require attributed device bytes within ±1%.
8. When loss is induced, require degraded coverage in the UI within two seconds.

Do not claim 10 Gbit/s support for a hardware profile until this gate passes.

## Per-device billing validation <a href="#per-device-billing-validation" id="per-device-billing-validation"></a>

The desired 99% accuracy means absolute byte error at most 1%, not 99% of packets delivered to the parser. This gate is not yet an achieved measurement. Run only against authorized test hosts and retain the reference counters, Monitor raw minute counters, exported Aegis points, allocation generations, configuration, sensor versions, timing, and quality/gap evidence.

1. Define the billed traffic first: Internet versus all traffic, receive versus transmit, byte layer (including which headers), GB/TB units, and accounting interval. Choose an independent unsampled reference at the same boundary; iperf application payload alone is not comparable with Ethernet frame bytes.
2. Generate known concurrent traffic for several allocated IPs, keeping each address's RX and TX reference separate. Include IPv4/IPv6, TCP/UDP and other IP protocols, small/MTU/jumbo frames, single VLAN and QinQ, fragmentation, snaplen truncation, ordinary traffic and peak packet rates. Exercise NIC offloads in the actual production configuration, and run the normal scanner concurrently in an approved phase to expose resource contention.
3. Check each direction of each device individually, not only the site total: `absolute_error = abs(observed_bytes - reference_bytes) / reference_bytes`. Require `absolute_error <= 0.01` for every nonzero reference over the agreed window, with no unexplained gaps. For a zero reference require zero exported bytes, rather than dividing by zero. Also inspect shorter windows for errors that cancel over a long run. Never scale observed bytes by packet retention.
4. Reconcile durable host minute bytes to Aegis minute rates multiplied by exactly 60 seconds and divided by eight. Verify duplicate delivery/replay does not increase totals. Do not substitute application totals or integrate live rates across unobserved time. Account explicitly for assignment-boundary minutes excluded from export and multiple addresses belonging to one device. For upgraded sensors, also compare Devices WAN cards and graph byte sums to the same canonical rows and time window. Verify `host_device_key` survives chunking/replay, multiple IPs are deduplicated before summing, A→B→A ownership changes do not transfer historical usage, and one busy IP cannot hide a gap in another IP's measurement. Legacy/unowned rows must remain unavailable on Devices even when a current IP binding or MAC WAN total exists.
5. Exercise mirrored captures, asymmetric/disjoint paths, migration, interface recreation, sensor restart, upload outages and backlog replay. Loss/conflict must remain uncertain, missing data must remain a gap, and unsent data must not silently become a complete zero. A failed accuracy case requires a resolved measurement boundary, recovery path, or reconciliation policy; a null coverage marker does not make the byte measurement pass.
6. Repeat over a representative sustained interval (at least 24 hours), with per-device cumulative reference counters and production-like load. Record worst-case directional error and every excluded/uncertain interval. Do not claim 99% for a topology or hardware profile until these results support it.

Packet counts cannot bound byte loss because missing packets have unknown sizes, and zero libpcap drops cannot expose traffic the selected capture never saw. Current strongest-capture selection cannot recover arbitrary disjoint paths or traffic during capture downtime. A long-term billing ledger and per-customer unsampled boundary counters are separate work, not supplied by this monitoring feed. See [Linux interface counter semantics](https://docs.kernel.org/networking/statistics.html) and [libpcap statistics limitations](https://www.tcpdump.org/manpages/pcap_stats.3pcap.html).

## Continuous port scanner canary <a href="#continuous-port-scanner-canary" id="continuous-port-scanner-canary"></a>

Use one authorized representative Proxmox hypervisor before fleet rollout. The default resource profile is intended for approximately 200 running VMs, but targets are IP addresses: dual-stack or multi-address guests count more than once. Do not run synthetic load against customer VMs without a suitable test window and authorization. Unit tests check limits and state transitions, not actual throughput, CPU consumption or guest latency.

1. Record a comparable baseline with the scanner service stopped: aggregate host CPU, available RAM, memory/CPU PSI, guest latency, bridge packet rates, disk pressure, and the normal monitoring agent's capture drops/telemetry latency.
2. Start only the canary's upgraded `l1-monitor-port-scan.service`. Verify its actual delegated subtree. Verify separate `supervisor` and `scan-<UUID>` leaves, their settings (`cpu.max`, `memory.high`, `memory.max`, `memory.swap.max`, `pids.max`) and that the normal agent is in a different service cgroup. There must be no aggregate one-CPU / 512 MiB limit at the scanner service root. The scanner should refuse to probe if limits or delegation cannot be verified.
3. Observe at least one complete sweep, including scans that continue beyond an hour. Record `cpu.stat`, `memory.current`, `memory.peak`, `memory.events`, `pids.current`, and actual state-directory file sizes including SQLite WAL. Track host metrics and guest latency against the baseline, not just scanner process usage.
4. Verify each host's CPU allowance is at most one logical CPU over its quota period, memory high/max are 384/512 MiB, and swap is disabled. At most eight daily full, two customer full and two common scans may run concurrently; every active IP has a separate leaf. The supervisor also has its own budget. In an authorized test, confirm a leaf OOM affects only that host, its result reports the failure/partial work, and other scans and normal telemetry keep running. Check normal restart and supervisor SIGKILL cleanup: no orphaned Nmap processes or stale cgroup leaves may block the next invocation.
5. In a controlled test, cross a pressure threshold and check that new work is deferred, active probes stop, and resumption needs 30 seconds of healthy measurements. Do not stress a production host merely to trigger this check.
6. Check 200-target scheduling and coverage, including IPv6 if used. Hourly common jobs request the top 100 TCP and top 100 UDP ports; daily and customer jobs request ports 1–65535. Set the site's daily release time and verify its IANA timezone, DST transition behavior, and one catch-up cycle after downtime. When both profiles are due, the common attempt precedes that IP's daily full attempt. Customer requests must use their dedicated pool, retain their UUID through retries, and never resolve from common-port results. Verify that scans cross schedule boundaries without being killed or overlapping another scan of the same target. Incomplete attempts stay partial and stale evidence keeps its original age. Missing results are not proof of closed ports. With sensor 0.1.24, verify the Operations Manual/Automatic Jobs tabs contain durable queued full jobs before capacity opens. Claiming a job must not imply Running until Nmap actually spawns. Verify start/heartbeat delivery, an overdue-contact indication during an outage, and terminal outcomes that survive newer hourly scans. Test result-before-start delivery and restart during a long phase: late proof may fill actual timestamps, never reopen a completed job or repeat its UUID. Filtered/silent-port sweeps that finish their checks should be Complete with uncertainty; interruption and missing output remain Partial, and neither condition claims old ports closed. Include controlled services with known software versions on standard and nonstandard ports (for example SSH on 2222 and HTTPS on 8443). Confirm the identification phase uses `-sV --version-all`, its product/version/probe confidence reaches Applications, and hidden versions remain unknown. Measure the added service-probe traffic and scan duration; the discovery packet-rate ceiling does not cap that traffic. Preserve Nmap's default safety exclusions. Compare known open ports and service fingerprints with a slower reference scan in the same test window, including a busy guest and a guest with rate-limited closed-port replies. Verify the tuned `-T4` policy (50/100/500 ms minimum/initial/maximum probe timeouts, one retry, 256 outstanding probes) improves discovery duration without an unacceptable loss of known services. Test both full and top-100 profiles and IPv6 where deployed. Record TCP and UDP discovery durations separately from `--version-all` identification; the latter's service waits are not shortened by the discovery RTT bounds.
7. Stop the scanner and investigate if guest latency, capture loss or host pressure rises unacceptably against the baseline. Do not increase limits merely to force completion. Roll out more widely only after the measured impact and achieved coverage are acceptable for that hardware.

The 8,000 packets/sec per-host ceiling allows up to 96,000 packets/sec at twelve active hosts. This covers port discovery, not every packet generated by service identification, replies, or guest/bridge processing. Twelve host budgets can allow up to twelve logical CPUs and 6 GiB of scan RAM, plus the separately bounded supervisor. Measure actual effects before fleet rollout; these ceilings are not a prediction of resource use or completion time. See [Nmap's timing controls](https://nmap.org/book/man-performance.html) and [Linux cgroup resource controls](https://docs.kernel.org/admin-guide/cgroup-v2.html).

## PostgreSQL storage snapshot <a href="#postgresql-storage-snapshot" id="postgresql-storage-snapshot"></a>

Run [postgres\_storage.sql](https://github.com/LayerOne-LLC/Network-Monitor/blob/master/tests/performance/postgres_storage.sql) from the repository root using an existing libpq connection configuration (`PGSERVICE`/`PGPASSFILE`, or the usual `PGHOST`, `PGDATABASE`, and `PGUSER` settings). Let `psql` prompt for a password if needed; do not paste credentials into the command or shared output.

```sh
psql -X -v ON_ERROR_STOP=1 -f tests/performance/postgres_storage.sql
```

The report uses a read-only transaction with statement and lock timeouts. It reads catalogs, statistics, and relation sizes; it does not scan telemetry rows or run maintenance. Table-family totals include all daily partitions, split into heap, auxiliary/TOAST, and index space. The remaining sections show large indexes, write/dead-row estimates, and partition bounds to check retention.

Capture before/after snapshots over comparable traffic windows. Row estimates and update/index-scan counters are approximate or cumulative, and reset times matter. They do not measure reclaimable disk space or prove an index is unused. The database-size total also includes objects outside the monitoring tables. See PostgreSQL's [size functions](https://www.postgresql.org/docs/current/functions-admin.html#FUNCTIONS-ADMIN-DBSIZE) and [statistics documentation](https://www.postgresql.org/docs/current/monitoring-stats.html).


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.layeronecloud.com/network-monitor/performance.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.