> For the complete documentation index, see [llms.txt](https://docs.layeronecloud.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.layeronecloud.com/platform/vps/operator-management.md).
# Unmanaged VMs and operator server management
Operators can assign an existing Proxmox QEMU VM to a client from Services → Add unmanaged VM, the client's Add unmanaged VM action, or Hypervisor guests → Assign to client.
## 5.14 Unmanaged client VMs
Operators can assign an **existing** Proxmox QEMU VM to a client from Services → **Add unmanaged VM**, the client's **Add unmanaged VM** action, or Hypervisor guests → **Assign to client**. The form takes account, cluster and VMID. Assignment reads the live guest, config and power state and creates a LayerOne ownership record; **it sends no Proxmox mutation and creates no order, subscription, invoice, deployment job or IP allocation.**
Import rejects templates, containers, locked or transitional guests, closed accounts, identity-verification holds, and VMIDs already owned or reserved. Disabled placement on a registered host does not prohibit importing an existing guest there. Ownership is checked again under the cluster allocator lock and the account lock. **Credentials from the guest configuration are not imported** — only selected identity/resource fields and observed NIC baselines. Imported addresses are not automatically claimed in IPAM.
`VirtualMachine.is_unmanaged` is a **persistent billing exemption**: these records have no plan, order, subscription or hourly price override, enforced by model validation **and a database constraint**. Existing VMs default to the normal billed mode. **The service editor and Django admin cannot toggle the exemption**; this workflow does not convert an existing paid service.
Unmanaged VMs appear in the client's instances and the operator's services as **Unmanaged · No billing**; retain owner-scoped power, console, settings and supported management controls; produce no runtime or managed-service charges and contribute neither traffic nor earned instance credits to the paid transfer pool; are excluded from financial suspension, bandwidth holds, billing recovery and billing termination — including on accounts that also own paid services; do not unlock hourly billing/API eligibility or increase automatic recharge; cannot buy a plan upgrade or managed services, or use catalog recreation/reinstall (those deployment changes remain in Proxmox); keep their existing network configuration (new LayerOne private-network attachments are unavailable, including via direct service/API requests, so a missing IPAM address cannot cause an observed guest NIC to be overwritten); and retain firewall management in Proxmox (LayerOne firewall edits and group assignments are unavailable).
**Financial account suspension alone does not block the owner's power or console access.** Independent KYC holds and explicit manual service suspensions retain their authority, and clearing a KYC hold durably restores unmanaged VMs independently of the credit balance. Explicit owner/operator destruction remains a separate authorized lifecycle action — the exemption prevents **billing-driven** teardown. The worker rechecks the exemption before stale financial or catalog jobs act.
Assignment is **super-admin only**, POST/CSRF protected, audited as `vm.unmanaged_imported`. Support and clients cannot assign ownership; GET only renders the form.
## 5.14.1 Operator-created services
**Services → Create service** (`admin-client-service-create`) deploys a server onto a client's account without them ordering it. It is the third operator entry point beside the unmanaged import above and the custom order in [4.3](/platform/billing/model-and-catalog.md#id-43-orders-and-checkout), and it is the only one that both bills and provisions.
It runs the **same sequence checkout runs** — `create_order_intake` then `queue_provisioning_for_paid_order` — so the result is an ordinary service with an order, a subscription, a provisioning job, a runtime meter and a credit ledger it draws down hourly. The order is `source: admin_service_create` (not the checkout source, so the 20-second duplicate-collapse window does not apply: two console submits are two servers) with an `admin_created_service` block naming the actor, whether it is free, and why.
Two things are operator-only, both explicit:
* **`operator_override=True`** on `create_order_intake` waives exactly two refusals — the paused-sales stock switch and the depleted-credit deploy block. Those exist to stop a *customer* buying what they have not funded. Suspended, closed and identity-hold accounts are still refused, and every template, ISO and networking validation still runs. Self-serve checkout and the client API must never pass it.
* **A service can be free.** That is `VirtualMachine.hourly_rate_override` of `0.0000` — the field that already existed, so metering, the console and the API all keep reading `vm_effective_hourly_rate`. A free server is a normal billed-mode service with a plan and a record; it is **not** `is_unmanaged`, which is a different thing (an imported guest with no plan at all). A free service requires a typed reason, recorded on the order, the billing event and the audit trail (`vm.admin_created`).
Cadence is always hourly and the upfront amount is forced to `$0.00`, because a non-zero one would produce an unpaid order and no server — quoting is what the custom order path is for. Private networks are deliberately absent from the form: a VNet is account-scoped, so the choices only exist after a client is picked, and the service's own network page attaches one after deploy. A generated root password is never echoed into a flash message; it is read from the server page, which decrypts the column.
**Free does not exempt the account from credit automation.** A free server is still a non-unmanaged VM, so an account whose balance reaches $0.00 still runs the grace window and suspension in [4.10](/platform/billing/credit-automation.md#id-410-credit-automation-the-000-floor-grace-window-suspension-termination), and a free server is suspended with the rest. Use the unmanaged path for a true billing exemption.
**Who may reach it.** Absent from `SUPPORT_CONSOLE_ROUTES`, so default deny keeps Support out: it commits hardware and starts charging somebody, which is the credit-adjustment class of power. The Services list and the client detail page are Support-readable, so the buttons into it are wrapped in `{% if not console_nav_is_support_staff %}`. `create_client_service` re-checks `actor.is_admin` itself rather than trusting the view decorator.
## 5.15 Operator server management
**The console can run every client-area action on a customer's server, and it is a second set of views, not a relaxed ownership check.**
`/console/billing/services//` mirrors the client page: Start/Stop/Restart/ Shutdown, Open console, Usage, Upgrade, Reinstall OS, Recreate, Settings, the managed-services switch and the destroy panel — plus the facts the client page has no business showing: VMID, cluster/node, effective hourly rate against the plan rate, account balance, and links to the order, the provisioning job and the billing record.
| Route | Method | What it does |
| ------------------------------------------- | -------------- | ------------------------------------- |
| `admin-service-detail` | GET | The page |
| `admin-service-action` | POST | One of the four power actions |
| `admin-service-console-create` | POST | Mint an operator VNC session |
| `admin-console-session` | GET | The VNC viewer |
| `admin-service-reinstall` | GET/POST | OS picker, hostname typed back |
| `admin-service-upgrade` / `-upgrade-cancel` | GET/POST, POST | Plan picker, drop a scheduled upgrade |
| `admin-service-managed-enable` / `-cancel` | POST | The managed-services addon |
**Why a second set of views and not a relaxed ownership check.** Letting an admin through `_client_owned_vm_or_403` does not work: `AccessModeRouteMiddleware` bounces an admin-mode session out of the `/client/` URL space entirely, so the page could never load; and `record_audit_event` stamps a write made under `/client/` as `actor_mode=client`, so every operator change would have been recorded as the customer making it.
**The service layer is shared unchanged.** `request_virtual_machine_action`, `request_virtual_machine_reinstall`, `request_plan_upgrade`, `enable_managed_service` and `cancel_managed_service` all take an `actor` and hold their own locks and blockers, and the operator views call exactly those. Nothing about power, provisioning or billing is reimplemented, so there is no second copy to drift. Only three templates are shared, because their content is genuinely identical for both audiences (`client_service_reinstall.html`, `client_service_upgrade.html`, `client_console_session.html`), and their back link comes from the view as **`service_detail_url`** rather than a hardcoded `billing_public:` name.
**There is no admin view of another customer's Client Area pages, and that is deliberate.** `_request_is_billing_admin` (`apps/billing/views/common.py`) returns False for any request under `/client/`, so an admin opening another account's `/client/services//` gets a 403 like anybody else. Admin visibility lives under `/console/billing/`, which is where the audit trail, the secrets handling and `@admin_required` all are. **If you are here because an admin got a 403 on a customer's page, the answer is the console, not a bypass.**
**Who may reach it.** Nobody was granted anything by default: none of the routes above are on `SUPPORT_CONSOLE_ROUTES`, which is the intended answer for a page that can power-cycle, wipe, reprice and watch the screen of a server belonging to someone else. Support keeps the Services list and the read-only usage page, so the links into the new page from those Support-readable pages are wrapped in `{% if not console_nav_is_support_staff %}`.
Four independent gates, each covering a case the others do not: `@admin_required` (a signed-in customer is refused even though `@login_required` passes) → `StaffRoleRouteMiddleware` (refuses Support on the resolved route) → `AccessModeRouteMiddleware` (an admin in Client mode has a GET redirected and a **write** refused) → `user_may_access_console_session` ([5.11](/platform/vps/console-and-metrics.md#id-511-vnc-console)).
Deliberately left alone: **Suspend / unsuspend are not tiles.** `VirtualMachineActionType` has them and the service already lets an admin actor request them, but they are not part of the client surface this page mirrors, so adding them here would be a new capability rather than a relocated one.
## 5.16 Operator resource sync
**The console record editor's cpu/memory/disk reach the hypervisor on save, and the reboot in the message is the contract, not a workaround.**
`/console/billing/clients//services//edit/` owns `cpu_cores`, `memory_mb`, `disk_gb` and `bandwidth_tb`. Saving used to write four columns and stop, so Proxmox kept the old allocation while the portal and the API reported the new one — and the two only agreed again if somebody ran a plan upgrade, which is a *billing* change and moves the price. The page said "local record only", so it was not a lie; it made the one field group an operator reaches for to change a customer's hardware the one field group that could not change it.
* **Not a provisioning job.** `RESIZE_VM` stops and starts the guest, and an operator correcting a number did not ask to reboot a customer's server. The push is `proxmox_adapter.sync_virtual_machine_hypervisor_resources`, a direct `PUT .../config` in the pattern `sync_virtual_machine_hypervisor_name` established for a rename.
* **The reboot is the contract.** Proxmox parks a live VM's `cores`/`memory` in its pending config and applies them at the next full stop/start: the hypervisor is correct immediately, the guest catches up on the customer's next power cycle, and the success message says so.
* **Only the fields that moved are sent**, so a save cannot quietly re-assert something else about the guest.
* **Disk grows immediately, filesystem does not.** `PUT .../resize` grows the block device on a running guest, but nothing inside notices until growpart/pvresize/lvextend run and cloud-init's growpart does not understand LVM. `_mark_guest_disk_grow_pending` writes `pending_guest_disk_grow`, and `_queue_vm_lifecycle_job` reads it: a **START or RESTART** sets `request_payload["grow_root_volume"]` and `_step_definitions_for_job` appends the existing `grow-disk` step. `agent=1` rides along with a grow, because the expansion runs over the guest agent and a guest cloned before that step existed has none enabled.
* **STOP and SHUTDOWN do not consume the marker** — the filesystem can only be grown from inside a running guest.
* **The marker is cleared at finalize, not at queue time**, so a job that failed before reaching the step leaves the work outstanding for the next power-on.
* **A pending plan upgrade still wins**: `_queue_vm_lifecycle_job` branches to `_queue_vm_resize_job` first, and the resize step list already ends with `grow-disk`.
* **`bandwidth_tb` reaches nothing, by design.** Transfer is pooled per account by `bandwidth.account_allowance_tb`; that column is a display figure that bills nothing, and the fieldset description now says so.
Traps already paid for:
* **The disk decision is made against what Proxmox reports, not the row's previous value** — the row is the thing that can be wrong. The three-way decision is the provisioning gate's (`_root_disk_resize_decision`): skip a size already within a GiB, refuse a real shrink, otherwise PUT. The root disk key is discovered from the live `boot` order (`_root_disk_key_from_config`), because appliance images boot from `virtio0` or `sata0`, not `scsi0`.
* **A shrink is a warning, not a failure.** The ModelForm has already saved and Proxmox will never make a disk smaller, so the honest report is that the record and the guest now disagree, naming both sizes.
* **The resize UPID is waited on, briefly.** An accepted-then-failed task (no space on the storage, a volume with snapshots) is precisely the case that left every indicator green on a server that never grew. `_raw_wait_for_task` polls for `OPERATOR_RESIZE_TASK_WAIT_SECONDS` (15); `_wait_for_task` cannot serve here because it logs every poll against a `ProvisioningJob` and waits ten minutes. A task still running when the budget is spent is **not** reported as a failure.
* **Nothing raises.** `sync_edited_virtual_machine_resources` swallows and logs, because the database write has already committed and a Proxmox blip must not 500 the page. Saving again re-issues the same idempotent calls.
* **A guest not on the hypervisor yet is skipped, not failed** (the same `_hypervisor_name_missing_is_ok` guard as the rename): mid-clone or mid-reinstall, the job about to run builds its config from this row.
* **Gated on `BILLING_PROVISIONING_LIVE`,** so a staging copy of a production cluster config cannot resize a live guest.
One `BillingEvent` per edit (`vm_resources_edited`, PROVISIONING category) carries the before/after pair for every changed column **and the hypervisor outcome**, because "saved" and "applied" are different claims and only the second is worth anything at 3am. `billing:admin-client-service-edit` is off `SUPPORT_CONSOLE_ROUTES`.
**Debug status override.** Operator **Settings** groups the status selector under **Debug feature** → **Manual server status**. An admin can select Running for a server recorded Failed or Stopped; the help explains this corrects the local record and **does not send a power command or repair the server on Proxmox**. Actual changes record `vm_status_manually_changed` with operator, server ID, and previous/new status; an unchanged save creates no event. The selector exists only on operator Settings — client settings do not accept status updates, and Support and Client-mode writes remain refused.
## 5.17 Compute host workspace
**Compute hosts** opens on the host directory: host, connection, live CPU/RAM usage, placement status, active service count, console access and editing. Clicking the host opens its metrics workspace; editing stays a separate row action. Search, connection/status filters, 50-row pagination. **Summary counts describe the whole recorded fleet, independently of filters.** "Enabled for placement" counts active, enabled hosts on enabled connections — it is **not** a live connectivity or provisioning-readiness check, and placement status remains saved configuration, separate from observed connectivity. Active services use the existing placement-load statuses (provisioning, staged, running, stopped). **View services** opens the exact host's inventory including suspended or failed services, and hosts with the same name in different clusters cannot mix their lists.
Separate sections (each its own `?tab=` URL, contextual action and inventory): Connections, Storage, ISOs, OS templates, Console settings. Resource inventories are searchable and paginated; storage, ISOs, and templates can also be filtered by connection. The OS templates directory lists active images unless Status is Inactive or All. Within each OS family the directory lists the highest version first (numeric runs in the template name). OS templates are versioned images grouped under `OperatingSystemFamily` tiles (bundled Debian, Ubuntu, Rocky Linux, AlmaLinux, Alpine Linux, FreeBSD, Windows and pfSense icons, with optional uploaded replacements). **Delete removes the local approval, not the source VM on Proxmox**, and is refused while any non-destroyed server still uses the image; destroyed servers drop the image link, and a product default pointing at it is cleared. **Credential presence is labelled Configured, not a claim that a live connection test succeeded.** Console settings configures customer VM VNC access and is distinct from the per-host root console.
**VMID reservations** live under Hypervisor guests → VMID reservations (`?tab=reservations`): a read-only directory with search, connection filtering, pagination, collision alerts, quarantine dates and verification history. It reads local reservations **without making a live inventory request**, and collision counts cover all connections even when filtered. The six-month quarantine rules in [5.7](/platform/vps/lifecycle.md#id-57-reinstall-recreate-destroy-and-vmid-quarantine) are unchanged.
**Host metrics.** The directory loads current CPU utilization and count, RAM utilization and used/total, uptime and observed connectivity. Missing saved capacity no longer prevents displaying actual hardware, and **saved placement configuration is not overwritten by monitoring reads.** The host workspace at `servers/nodes//` adds CPU, memory, swap, root filesystem usage, I/O wait, three load averages, uptime and host software/CPU identity, and shows the platform's assigned service counts and allocated resources **separately** from measured usage (those totals are scoped to the exact local node, exclude destroyed records, and are not a census of unmanaged guests). History (CPU, RAM, network rx/tx, I/O wait, load) comes from Proxmox RRD for hour/day/week/month/ year; **missing samples remain gaps and zero is a valid reading**. The root filesystem metric describes the host root filesystem, not every guest storage pool.
HTML renders without waiting for a remote request. `servers/metrics/` accepts at most 50 local node IDs and groups observations by connection; `servers/nodes//metrics/` serves the detail snapshot and selected history. The browser refreshes every 30 s, pauses in hidden tabs and ends polling on navigation. Read caches are short-lived; HTTP responses are `no-store`. **A failed refresh cannot silently leave an old reading labelled current**, and a reported offline host is distinct from an unreachable API or a disabled connection.
**Host shell sessions are single-use.** The relay atomically claims the session after authenticating its owner, preventing a second websocket from opening another root terminal with the same token. Closing the tab, leaving the shell page (including console navigation) or losing the connection ends the session and closes its upstream socket. The relay also stops the exact Proxmox `vncshell` task returned by `termproxy`, including when upstream connection or authentication fails, and retains a pending ticket response during cancellation so a late-created task can still be stopped. **The task's UPID identifies its worker node, which may differ from the requested host** when the API connection tunnels through another node; cleanup never targets unrelated host sessions. An open shell expires at its original 20-minute deadline; if the browser stops sending terminal traffic without a disconnect event the relay closes it after 120 s, and the upstream authentication handshake has a 15 s limit. Switching browser tabs does not close a shell. Session rows remain as ended history with their revocation time; **tickets are never persisted**. Deliberately detached commands are outside the session's lifetime and are not killed by a host-wide sweep.
All these views require admin access and stay outside the Support allowlist. Metrics GETs queue no jobs, update no configuration and modify no hypervisor. Authenticated reads are independent of the provisioning-write switch. Secrets and raw upstream errors never enter the response.
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:
```
GET https://docs.layeronecloud.com/platform/vps/operator-management.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.