> For the complete documentation index, see [llms.txt](https://docs.layeronecloud.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.layeronecloud.com/platform/vps/console-and-metrics.md).
# VNC console, usage metrics and NIC inventory
ConsoleSession records a hashed token, expiry, owner, request IP, user agent and VM metadata.
## 5.11 VNC console
`ConsoleSession` records a hashed token, expiry, owner, request IP, user agent and VM metadata. Sessions are short-lived and store only a token **hash**. Raw Proxmox credentials and direct VNC ports are never exposed to the browser; the broker creates Proxmox VNC tickets server-side and proxies noVNC traffic.
`create_console_session` takes `admin=True` for the operator path, which drops the ownership check and the live-billing-record check (an operator diagnosing a server whose arrangement was already cancelled is exactly the case the client-side gate exists to refuse). **The flag is not taken on trust** — the actor must satisfy `is_super_admin`.
The row records `user=actor`, and **that** is what authorises the viewer. `user_may_access_console_session` recognises **exactly two principals**: the billing-account owner (via `user_can_access_tenant`, see [1.10](/platform/identity/organizations.md#id-110-organizations-and-tenants)) and the operator the session was minted for. Being staff is not enough — a second super admin handed the URL gets a 403.
**That function is deliberately stricter than the endpoint around it, and asks for `is_super_admin` rather than `is_admin`, because it is the entire gate on the websocket.** `_handle_console_websocket` is dispatched by the ASGI wrapper *before* Django, so `StaffRoleRouteMiddleware`, `AccessModeRouteMiddleware` and `@super_admin_required` never see the socket. Every principal refused on the HTML page has to be refused again in that one function.
`_CONSOLE_WS_PATH` matches both mounts (`/client/console//ws` and `/console/billing/console//ws`): one relay, two URLs, authorised from the session row rather than from the path. Connected relays recheck the persisted session, the active user and tenant authority **every 10 seconds** with a five-second lookup timeout; revocation, expiry, lost membership or a lookup timeout closes both directions at that check.
**The operator console URL carries a live credential in its path**, so `/console/billing/console/` is registered in **both** `apps/core/credential_paths.py` (with a redaction rule) and `apps/core/error_reporting.py`. Without the first, the token lands in the request trail, the analytics `dl` parameter and the audit log; without the second, in a `ServerErrorReport`.
Console session creation writes a billing event. Console settings (customer VM VNC access) are configured under Compute hosts and are distinct from the per-host root console ([5.17](/platform/vps/operator-management.md#id-517-compute-host-workspace)).
The browser viewer (noVNC) **scales the framebuffer locally** to the console pane and does not ask QEMU to resize the guest desktop. Requesting a remote desktop size that the guest does not actually apply leaves the USB tablet calibrated to a different rectangle than the pixels on screen, which is the failure mode where the pointer only tracks through part of the display.
## 5.12 Usage metrics, RRD history and disk I/O pressure
**Live rates come from Proxmox RRD, not from a cross-request delta.** `apps/billing/vm_rrd.py` fetches `GET /nodes/{node}/qemu/{vmid}/rrddata?timeframe=…&cf=AVERAGE`, where each point carries `cpu` (0..1 **already normalised to the guest's allocated vCPUs**), `mem`/`maxmem` in bytes, and `netin`/`netout`/`diskread`/`diskwrite` already averaged to bytes per second. Cached 60 s per VM+timeframe, successful reads only. `five_minute_io_averages` takes the mean of the last five minutes of the hour timeframe; per-key nulls are skipped and an all-null window reports as no data.
`live_vm_metrics_snapshot` (`apps/billing/vm_metrics.py`) takes network and disk I/O rates from those five-minute averages, so **the first poll already has values**. The old counter-delta path remains only as a fallback when the RRD read fails (its baseline cache is still written to keep the fallback warm). The previous design cached the previous poll for 120 s in Redis shared with the Celery broker, so the tiles could sit on "Measuring…" forever.
`_cpu_percent` does **not** divide by the vCPU count: Proxmox already normalises `cpu`, and the old formula under-reported CPU by the core count. Stored `VirtualMachineMetricsSample.cpu_percent` rows written before that fix carry the smaller values.
`vm_usage_context` builds six SVG polyline charts (CPU %, memory %, network in/out, disk read/write) plus range aggregates from the RRD series, with ranges mapped to Proxmox timeframes (hour/day/week/month/year). **Transfer totals are the mean rate times the covered seconds — an estimate for context, not metering** (metering stays with the bandwidth poller). Client: `/client/services//usage/`; admin: `/console/billing/services//usage/` with a `admin-service-stats` JSON endpoint feeding the live tiles. The live metrics panel and its polling script are one shared partial parameterised by `stats_url`.
Chart rate formatting uses **bytes per second** (`MB/s`); the admin aggregate network chart previously used a bits-per-second formatter and under-labelled by 8×.
**Disk I/O pressure.** The platform could show how many **bytes** a guest moved and nothing about how many **operations** it took, and those are not the same measurement:
| Workload | Throughput | Operations | Effect on the array |
| ------------------------------ | ---------- | ---------- | ------------------- |
| Backup, 4 MB sequential writes | 200 MB/s | \~50/s | negligible |
| Database, 4 KB random writes | 200 MB/s | \~51,000/s | saturated |
Three sources, one attribution:
1. **Real IOPS from QEMU block accounting** (`vm_blockstats.py`): `rd_operations`, `wr_operations`, `rd_total_time_ns`, `wr_total_time_ns`, `flush_operations`, `flush_total_time_ns`. Preferred path is `blockstat` on the **existing** `status/current` response (PVE builds that body with the full form of `vmstatus`, which runs `query-blockstats` for us — structured JSON, no extra privilege, **zero extra API calls**, since the bandwidth poller already makes this request every sweep). Fallback is `POST /nodes/{node}/qemu/{vmid}/monitor` with `info blockstats`, parsed from human-monitor text: one extra request per VM and it needs `VM.Monitor`. The time counters are the valuable half — `rd_total_time_ns / rd_operations` over a window is the **mean service time of one request**, which is as close to per-guest I/O wait as a hypervisor can get.
2. **Node I/O wait** (`node_metrics.py`): `wait` from `GET /nodes/{node}/status`, history from the `iowait` RRD series. This is the **only** direct measurement that the array itself is behind.
3. **Attribution** (`io_pressure.py`): a node is **under pressure** only above `NODE_IO_WAIT_HIGH_PERCENT`; below it nothing is attributed to anyone, because a guest at 20,000 IOPS on an array serving 200,000 is a customer using what they bought. On a pressured node a guest over `VM_IO_DOMINANT_SHARE_PERCENT` of all guest operations is a **cause**; one guest at 40% did this, while ten guests at 10% each are a capacity problem and naming any one of them would be wrong. High latency with a small share is a **victim**.
Things that will bite whoever changes this next:
* **Ranking by latency gets the blame backwards.** A guest issuing a handful of requests into a saturated queue waits *longer* than the guest saturating it, so sorting by "who is waiting most" puts the victims on top. The tables rank by **offered load**; the latency ranking is still rendered, deliberately labelled as such.
* **Node I/O wait is denormalized onto every VM sample, on purpose.** Pairing two independent time series means matching a VM reading with whichever node reading is nearest in time, and under contention a mispairing of one poll interval inverts the answer. The poller reads each node once per sweep, **before** the guest sweep, and stamps that figure onto every VM sample from the same sweep.
* **The node list is built from node samples, not guests.** A host whose guests failed to sample still has a `ProxmoxNodeMetricsSample`; iterating `guests_by_node` made that host vanish, zeroing `io_pressured_node_count` at the moment the array was saturated.
* **`None` is never zero.** An unmeasured window, a guest with no block accounting, and a guest that issued no requests are three different statements, and all three would render as "0" — where zero IOPS reads as "innocent" and zero latency ranks an idle server as the healthiest thing on the node. `disk_accounting_available` is its own recorded column rather than inferred, because a freshly booted guest legitimately reports zero operations.
* **A reset counter means no *****rate*****, but full *****accumulation*****.** A guest reboot returns QEMU's counters to zero. Rates refuse that window (`_window_delta` → `None`), because a rate across a discontinuity feeds a verdict that names a customer. Lifetime totals do the opposite (`accumulation_delta` credits the full current value), because those operations really did happen — and discarding them would open a way to hide, since a guest rebooting in a loop would accumulate nothing while hammering the array continuously.
* **Only column-zero lines in `info blockstats` are guest drives.** Indented lines describe the images *behind* a device (backing chain, PVE throttle filters) and repeat the same field names; summing them multiplies a guest's IOPS by the depth of its image chain.
* **Latency is microseconds.** NVMe answers in tens of microseconds, so milliseconds round every healthy guest to "0ms"; nanoseconds put six meaningless digits in front of the operator. The `io_latency` filter and the panel's `formatLatency` must stay in step.
* **Proxmox keeps no operations series at all**, so the IOPS history charts read `VirtualMachineMetricsSample` and reach back only `VM_METRICS_RETENTION_DAYS`. `vm_io_history_context` is therefore separate from `vm_usage_context` with its own range selector — sharing the RRD's selector would silently return empty charts for the longer ranges.
**Abuse retention.** `VirtualMachineIoProfile` is a durable per-server record, because the two things that make the sample table cheap make it useless for abuse work: samples are pruned and cascade off the VM. The pattern that matters most — spin up a server, saturate the array, destroy it, repeat — therefore left *less* evidence the more times it was repeated. The defence is to **never re-read the samples**: every field is folded forward as each sample is written, so the record is complete before the samples are pruned. Identity is denormalized (`virtual_machine`/`billing_account` are nullable `SET_NULL`, with `vm_label`/`account_label`/`vmid`/`node_name` carrying who it was). **The signal is duration, not peaks**: `high_iops_sample_count` and `high_latency_sample_count` count polling intervals over threshold, because every nightly backup peaks and what separates a busy database from a torrent box is how long it sustains. **Reviewing mutes, it never clears** — a legitimately busy database should leave the queue, but its history is the record needed next time. The account rollup is the anti-churn view.
Settings (they are settings, not constants, because the right number is a property of the hardware — 1,500 IOPS is a quiet afternoon on NVMe and a saturated array on spinning disks):
| Setting | Default | Meaning |
| ------------------------------ | ------- | ------------------------------------------------- |
| `VM_IO_HIGH_IOPS_THRESHOLD` | 1500 | Combined read+write ops/s marking a guest heavy |
| `VM_IO_HIGH_LATENCY_US` | 20000 | Mean service time marking a guest as waiting |
| `NODE_IO_WAIT_HIGH_PERCENT` | 10.0 | Host I/O wait at which the array is judged behind |
| `VM_IO_DOMINANT_SHARE_PERCENT` | 40.0 | Share of node operations before attribution |
| `VM_IO_ABUSE_REVIEW_INTERVALS` | 12 | Over-threshold intervals before review (\~1 h) |
Access: `admin-io-pressure` is **Support-readable (GET only)** — infrastructure observability of the same kind as the usage monitor. `admin-io-abuse` and `admin-io-profile-review` are **deliberately absent** from `SUPPORT_CONSOLE_ROUTES`: the abuse page presents customers as suspected abusers and is where enforcement decisions come from.
## 5.13 NIC hardware inventory
Proxmox guests have a logged **hardware identity** for their NICs. One `VirtualMachineNic` row per `netN` slot: slot, display name (`Public` or the private VNet name), kind (public/private/unknown), **logged MAC (the baseline)**, last observed MAC, bridge, virtio model, present/removed. Guest OS names (`eth0`, `ens18`) are not collected — those need an agent.
Jobs: **daily 05:15 Eastern** `billing.collect_vm_nic_hardware` finds pollable VMs with no logged MAC and `GET`s qemu config to fill them — it does **not** overwrite a logged MAC and does not alert. **Weekly Monday 05:45 Eastern** `billing.verify_vm_nic_macs` re-reads every logged guest; if a logged MAC no longer matches, or a logged NIC has vanished, it opens a `VirtualMachineNicAlert` and emails super admins one digest.
New deploys and reinstalls **re-baseline** after the job succeeds (expected MAC changes, no alert). Attaching a private NIC fill-logs the new slot; detaching marks the slot absent without alerting.
The owning client may read the baseline on their Network tab per present interface. An interface without a baseline says **Pending collection** — the UI never invents a MAC. The client surface does not expose `last_observed_mac`, drift alerts, fleet search or baseline-accept controls; those live on the operator Network tab and at Customers → Services → **Hardware** (`billing:admin-service-hardware`, off `SUPPORT_CONSOLE_ROUTES`), where operators can collect missing MACs immediately, check one guest now, or accept a drifted MAC as the new baseline.
**MAC lookup.** The admin Instances search matches both the logged baseline and the last observed MAC on every recorded NIC, accepting colon, hyphen, dotted and compact notation case-insensitively, and colon-separated fragments; matching several NICs returns the instance once and preserves the existing filters. Hypervisor guests reads NIC config for **every** QEMU or LXC guest reported by each configured connection — including stopped guests, templates, untracked guests, nodes without a local host record, and hosts disabled for placement. These are read-only config requests made when loading the inventory, with at most **eight concurrent config reads per connection**, a five-second per-read timeout and a fifteen-second collection budget per connection, distributed across hosts. **Only extracted MACs are cached** (five minutes); raw config and secrets are never cached. Failed reads retain only a safe error for sixty seconds. Guests skipped when the budget expires stay eligible next request; cache failures fall back to live reads. **Unavailable MACs keep their guests visible and warn that MAC search may be incomplete.** The live inventory creates no billing records and changes no baselines.
NIC parsing accepts the QEMU model/MAC shorthand, explicit `model`/`macaddr`, and LXC `hwaddr` in any option order.
**What this is not:** not a guest-agent inventory, and **not a lock that prevents Proxmox edits — it detects them.**
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:
```
GET https://docs.layeronecloud.com/platform/vps/console-and-metrics.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.