> For the complete documentation index, see [llms.txt](https://docs.layeronecloud.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.layeronecloud.com/platform/vps/lifecycle.md).
# VM lifecycle, upgrades and rebuilds
request\_virtual\_machine\_action is the single service for start, stop, restart and shutdown, called by the client area, the operator console and the API with an explicit actor.
## 5.5 VM lifecycle actions
`request_virtual_machine_action` is the single service for start, stop, restart and shutdown, called by the client area, the operator console and the API with an explicit `actor`. It queues a `VirtualMachineAction` and a lifecycle `ProvisioningJob`; live power mutations follow the provisioning switches.
That `VirtualMachineAction` row is also what the portal renders "Starting" and "Stopping" from — Proxmox reports no such state, so the transition is ours. What a server's power state *is*, as opposed to what we asked it to become, is polled separately; see [5.22](/platform/vps/marketplace.md#id-522-power-state-polling).
* **`shutdown` asks the guest first and cannot be refused by it.** `status/shutdown` goes through the guest — Proxmox tries the QEMU guest agent, then ACPI — so on its own it held the job for the full timeout and failed it on `qga command 'guest-ping' failed - got timeout` whenever the agent was absent or the OS ignored ACPI. It is therefore always sent with `forceStop=1` and `timeout=GRACEFUL_SHUTDOWN_TIMEOUT_SECONDS` (120): the guest gets two minutes to close files and flush disks, and then Proxmox terminates the process itself (SIGTERM, then SIGKILL). `stop` stays the immediate hard cut — it is the control a customer reaches for precisely when the guest is not answering.
* **A forced shutdown is a success, not a failure.** Proxmox still ends that task unhappily, because it reports the agent it could not reach. `_recover_forced_shutdown` reads the guest's state back and, when it is `stopped`, clears the failure: the step keeps its **SUCCEEDED** status with no error written against it, and what actually happened is recorded on the step's `result_summary` (`forced_stop`, `graceful_shutdown_detail`) and in the action log. It trusts that state read rather than matching Proxmox's wording, which varies by version. A guest still running is a real failure and stays one.
* **`restart` is that same graceful shutdown followed by `status/start`**, waiting on the shutdown's task in between. Not `status/reboot`, which has no `forceStop` and so fails outright on a guest that ignores ACPI; not `status/reset`, which keeps the same QEMU process, so Proxmox never regenerates the Cloud-Init drive and a guest restarted after an IP change comes back on the old address — the exact thing the restart offered alongside an IP change exists to fix. `LIFECYCLE_PROXMOX_ACTIONS` is the one place these sequences are written; the unmanaged-guest buttons derive theirs from it, shortening only the grace period because that path runs inside the operator's own request rather than in a worker.
* **There is no QEMU pause as a customer action** — it does not survive a node reboot, Proxmox shows paused guests as running, and `start` on a paused guest is rejected as already running. A stopped server keeps billing.
* **A power action that is already the server's state queues nothing.** `redundant_power_action_message` polls the hypervisor (falling back to the VM row) before the row lock, and `request_virtual_machine_action` raises `RedundantPowerActionError` — a `ValueError` subclass, so every caller still refuses, but a distinct type the views render as a **warning**. No `VirtualMachineAction`, no lifecycle job, no BillingEvent: pressing Start twice on a running server is a customer refreshing a page, and the console's job list stays a record of work that was actually attempted. Restart is never redundant — it is a power cycle whatever state it finds.
* **Power actions are refused on a billing-suspended server**, for the operator too. Hiding the buttons was not enough: the action endpoint and the instances-list bulk form are both reachable by direct POST, and a lifecycle job that started the server also restored VNC and was never re-paused by the repair pass. Deletion stays available — **a customer must always be able to stop being billed for something.** The page renders `virtual_machine_power_blocker(vm)` and withholds the tiles with the reason stated.
* **Power actions are also refused while any provisioning job is live on the server.** A reinstall or resize owns the VMID for its whole stop/destroy/clone sequence; a start queued alongside it runs from a second worker against the same VMID.
* **Retiring a catalog template must not block power, resize, suspend or destroy.** `_selected_template_for_job` refuses an inactive `selected_proxmox_template_id` because that id is the clone source for a new deploy or a reinstall. Lifecycle jobs stay linked to the original order, so that check used to fail restart and shutdown after an operator disabled the OS the customer originally picked. `_plan_from_existing_vm` therefore only consults the catalog for `REINSTALL_VM`; every other existing-VM job uses `vm.template` as a label of what is already running.
* VM task failures use **customer-safe messages**, including for historical failed tasks. Raw provider errors remain on jobs and actions for operators; the admin task and service views show the customer message alongside the technical diagnostic. Private-network, firewall, floating-IP, reverse-DNS and console failures also hide provider detail in client HTML, API responses and console connection rejections.
**QEMU guest-agent password changes.** The Access section on client and operator overviews links to `services//change-password/` in the matching audience's URL space. The form changes the VM's configured login user (`effective_cloud_init_username`), including an existing Windows account; it does **not** accept an arbitrary username or create guest accounts. GET only displays the form and the latest persisted attempt. A CSRF-protected POST validates and confirms the password, locks the VM, rechecks active billing and service state, and queues a `change_password` job. **The pending password is encrypted in a dedicated job field** — never JSON payloads, action metadata, API logs or rendered form values. The dispatcher invokes Proxmox's native `POST /nodes/{node}/qemu/{vmid}/agent/set-user-password` with `crypted=0` (Proxmox handles the guest protocol's base64), which requires a running enabled guest agent, an existing account, a password the guest OS accepts, and `VM.GuestAgent.Unrestricted` on the token. There is no guest shell command, Cloud-Init rewrite, reinstall or restart. The worker rechecks the account and the immutable VM target before execution; **only confirmed live success** replaces the saved password via `set_root_password()`. A dry run or failure preserves it. Provider errors and response bodies are suppressed for this credential operation. The encrypted pending secret is cleared on success and retained on failure for explicit retry; an older attempt cannot be retried after newer VM work, and manual resolution cannot manufacture a successful password change.
## 5.6 Plan upgrades
A client picks a larger plan for one of their servers, the **billing rate moves immediately**, and the new vCPU/memory/disk are applied by a `RESIZE_VM` job that rides along with their next power-on.
**There is nothing to prorate.** Every plan on every cadence is converted to an hourly rate and debited from credit as the hour closes, and the rate is read from `vm.plan` at that moment — so repointing the plan *is* the entire money change. No `Order`, no `Invoice`, no `PaymentAttempt`, no ledger entry written by the upgrade itself. The in-progress `UsageRecord` stays mutable until its hour closes, so an upgrade at :55 bills that whole hour at the new rate; the maximum exposure is one hour of the rate difference.
`plan_upgrade_blocker(vm, plan)` (`apps/billing/client_services.py`) is the **single authority** on what counts as an upgrade. A target must be on the same `Product`, be active and not `hidden_from_catalog`, carry an `hourly_price` and an active `vm-runtime` `UsageMeter`, match `compute_family` **and** `network_rate` (resize never rewrites `net0`, so a Network Optimized server cannot move onto a General Compute SKU or the reverse), hold or grow **every** one of cpu/memory/disk/bandwidth, and grow at least one. `upgrade_target_plans_for_vm(vm)` filters the catalog through that same predicate so the picker can never offer a plan the service layer refuses, and `PlanUpgradeForm` builds its choices from the caller's eligible list so a forged `plan` id fails `invalid_choice`.
**The disk is a hard floor, not a policy preference:** Proxmox `/resize` refuses to shrink, so a smaller-disk target could never be applied at any price and is rejected at request time rather than discovered as a mid-job failure.
`request_plan_upgrade` repoints `vm.plan` and `subscription.plan`, writes the pending record to `vm.metadata["plan_upgrade"]`, and emits `plan_upgrade_requested`. It refuses a suspended account, an unprovisioned or wrong-status server, a negative credit balance, a server with an active provisioning job, and a second concurrent upgrade. **The spec columns deliberately do not move** — they keep describing the hardware that is actually running; only the plan (and therefore the price) changes at request time.
**Cancel is guarded on both sides.** `cancel_plan_upgrade` restores the previous plan on the VM and the subscription. "Only reachable while the resize is still outstanding" used to be an assumption, and it was self-serve exploitable: request the upgrade, click **Restart now to apply**, then POST cancel while the `RESIZE_VM` job sat in the queue — the plan and rate went back and the job wrote the larger cpu/memory/disk anyway, and Proxmox cannot shrink a disk, so that was upgraded hardware kept permanently at the old price. Now `cancel_plan_upgrade` raises when `_vm_has_active_provisioning_job`, **and** the RESIZE\_VM success branch applies the target specs only while the pending record is still `pending` *and* its `provisioning_job_id` is this job (`_resize_job_owns_pending_upgrade`). A job that no longer owns the upgrade finishes normally, moves only the power state, and records `resize_specs_applied: False`. **Two independent transactions guarding real money need two checks, not one.**
Applying: `_queue_vm_resize_job` intercepts `_queue_vm_lifecycle_job` when the action is START or RESTART and an upgrade is pending, creating a `RESIZE_VM` job keyed `resize:vm:{id}:action:{id}`. STOP and SHUTDOWN stay ordinary — the resize waits for the start that follows. Two step shapes chosen at queue time: `RESIZE_PROVISIONING_STEPS` (stop → verify-stopped → resize → start → verify) for a running server, `RESIZE_STOPPED_PROVISIONING_STEPS` for one starting from stopped. **Stop → resize → start is the only shape this adapter can verify:** raising cores or memory on a running VM returns 200 but lands in Proxmox's *pending* config unless hotplug is enabled, and the adapter never reads `/config` or `/pending`, so the job would report success while the customer kept the old size.
**A failed resize stays `pending` on purpose.** The plan and rate remain upgraded, so an admin retry is coherent and the customer's next start re-attempts it; reverting automatically would fight a retry and leave the job's target sizes pointing at a plan the server is no longer on. The client page surfaces the error next to an explicit **Cancel upgrade**.
**The order is never rewritten.** After an upgrade `vm.plan != vm.order.plan` permanently, which previously failed `VirtualMachine.clean()`. `has_plan_change` (on both `VirtualMachine` and `Subscription`, read from metadata) scopes the exemption to servers that really have been upgraded, and `ProvisioningJob.clean()` exempts `RESIZE_VM` specifically.
Intentional limits: **no downgrades** (storage cannot shrink, so it needs its own rebuild-or-migrate design; the pricing FAQ points downgrades at support), no email, no admin approval gate, no node capacity pre-check, and **bandwidth is compared but not billed** (the authoritative allowance is the account pool), so the upgrade UI does not advertise a transfer increase.
## 5.7 Reinstall, recreate, destroy, and VMID quarantine
**Reinstall** rebuilds onto the same VMID after cleanup: either a template clone (with SSH and disk-growth steps) or a blank ISO guest ([5.19](/platform/vps/images-and-backups.md#id-519-iso-library)). Hostname is typed back to confirm. Images whose minimum storage exceeds the server's disk stay in the picker; choosing one asks the customer to change to a plan with at least that much storage, and `request_virtual_machine_reinstall` refuses the POST so destroy-then-clone cannot leave the guest FAILED. `virtual_machine_reinstall_blocker` treats a catalog template **or** a ready account ISO on this cluster as sufficient; an empty template list must not hide the form when an ISO is available. Reinstall cleanup treats a **narrowly confirmed** missing Proxmox VM configuration as a warning, skips the remaining no-op cleanup and continues through clone, config, networking, boot, verification, guest setup and finalization, retaining the local service record and VMID; unrelated Proxmox failures remain blocking.
**Destroy** (`request_virtual_machine_destroy`) is idempotent when Proxmox reports the QEMU configuration already absent — but **a cluster-wide VMID lookup must confirm absence first**, and if the guest migrated the job retargets its current configured node. Destroy does not clone, so a later-retired catalog template does not block it ([5.5](#id-55-vm-lifecycle-actions)). Confirmed-absent guests skip the remaining remote-only steps, still release network resources, finalize local removal, and finish with a **warning** so the discrepancy stays visible without blocking cleanup.
One narrow local-only exception exists for queued `DESTROY_VM` jobs: a node-less VM record that has **never been provisioned** (`provisioned_at` is null) is a local placeholder, so the job makes no Proxmox calls, skips its remote steps, performs the normal primary-IP/PTR and private-network cleanup, quarantines a known cluster VMID, retains the history row as `DESTROYED` with `destroyed_at` set, and finishes with a warning. **A node-less record with a non-null `provisioned_at` still fails closed**: the missing assignment is not proof that the live guest and its data are gone.
**Deleted VMIDs are quarantined and checked daily.** Every successfully destroyed VMID, recreated destroyed service and operator local-record deletion writes a durable `VMIDReservation` **before** the local identity can be cleared. The reservation is scoped to the Proxmox cluster and lasts **six calendar months**. `billing.verify_reserved_vmids` lists cluster inventory once per cluster per day. A VMID is eligible for reuse only when **all** of these hold: the quarantine has elapsed; the most recent successful absence confirmation is ≤ 24 hours old; the latest check did not fail; and Proxmox did not report a guest using it. A remote match flags the reservation under **Hypervisor guests → VMID reservations** and keeps it out of the allocator; a failed or missed check also **fails closed**. When an eligible ID is finally claimed, any retained destroyed VM row has its VMID moved into metadata first, preserving the historical identity without violating live per-cluster uniqueness.
**Recreate** re-deploys a destroyed service from its catalog record. The operator control can also retry a failed provisioning job when an operator has deliberately left the VPS status as **Running**: it requeues the existing job and preserves its successful steps rather than creating a second job, and the same override is not exposed in the client portal.
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:
```
GET https://docs.layeronecloud.com/platform/vps/lifecycle.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.