> For the complete documentation index, see [llms.txt](https://docs.layeronecloud.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.layeronecloud.com/platform/vps/images-and-backups.md).

# ISO library, backups and snapshots

ISOs live on a shared Proxmox storage, typically l1-isos, not as durable application blobs.

## 5.19 ISO library <a href="#id-519-iso-library" id="id-519-iso-library"></a>

ISOs live on a **shared Proxmox storage**, typically `l1-isos`, not as durable application blobs. The connection field `iso_storage_id` names that storage. Leave it blank to hide the library on that connection. The Compute hosts **ISOs** tab catalogs files already on the storage (Refresh from Proxmox) and accepts an `.iso` file upload (Upload ISO). Support cannot open those operator routes. There is no customer download-URL ingest: Proxmox is never asked to fetch a URL the customer supplied.

**File ingest.** The public site sits behind Cloudflare, which returns **413** for a single request body over 100 MB on Free/Pro (200 MB Business; up to 5 GB on Enterprise) when the body is posted to this origin. Installer ISOs are larger than that.

The ISO library storage (`l1-isos`) is the rclone mount of a private bucket on the compute hosts. Super admins set that bucket on the Proxmox connection (Infrastructure / Connections): endpoint, bucket, region, object prefix, and write-only access keys stored with Fernet (`set_secret` / `get_secret` on `ProxmoxClusterConfig`; blank keeps the stored key). They are not environment variables. When the connection has a complete bucket config, `/import/start/` returns `mode: "direct"` plus presigned PUT URLs for the **final** ISO key (object prefix, default `template/iso`, plus the customer filename). The browser PUTs \~128 MiB parts (`ISO_UPLOAD_OBJECT_PART_MAX_BYTES`). It starts two transfers and raises concurrency while throughput is still climbing, then holds when an extra stream slows the upload. The cap is 8, or lower on small-memory devices, and never more than about 1 GiB in flight. The status line shows percent, bytes, and throughput, not how many streams are open. The upload page preconnects to the bucket host so the first PUT does not wait on DNS and TLS. Start does not wait on a server-side CORS probe; **Test ISO bucket** still checks CORS. Then it POSTs `/import/complete/` with the part ETags. Those PUTs never touch Postgres, the Railway origin, or Proxmox. After complete, a daemon thread (inline when dry-run or eager Celery) lists `GET /nodes/{node}/storage/{storage}/content?content=iso` until `{storage}:iso/{filename}` appears on the mount (rclone directory cache can lag; wait up to `ISO_UPLOAD_APPEAR_SECONDS`). Proxmox is not asked to `download-url` or `/upload` that object — the bytes are already on the filesystem the storage plugin reads. The object is deleted only when the import is discarded or the catalog row is deleted, never after a successful import. The library bucket is S3-compatible (Backblaze B2 in production). B2 dashboard “share with this origin” radios only allow **downloads (GET)**. Browser ISO ingest needs a **custom** CORS rule on the **S3 Compatible API** (or Both, not Native-only) with `s3_put`, `allowedHeaders *`, and `exposeHeaders ETag`. Native-only CORS does not apply to these PUTs. `b2 bucket update` **replaces** existing CORS rules; keep any download rules in the same JSON:

```json
[
  {
    "corsRuleName": "isoBrowserPut",
    "allowedOrigins": [
      "https://layeronecloud.com",
      "http://127.0.0.1:8000",
      "http://localhost:8000"
    ],
    "allowedHeaders": ["*"],
    "allowedOperations": ["s3_put"],
    "exposeHeaders": ["ETag"],
    "maxAgeSeconds": 3600
  }
]
```

```bash
b2 bucket update --cors-rules "$(<./iso-cors.json)" YOUR_BUCKET allPrivate
```

Listing objects from the server can succeed while the browser still fails: Firefox reports a blocked CORS PUT as `NetworkError when attempting to fetch resource.` Access keys never render back on the connection form and are not on `SERVICE_ENV_ALLOWED_KEYS`. **Test ISO bucket** lists that prefix with the stored keys, then sends an OPTIONS preflight for PUT from `https://layeronecloud.com` and the console origin. It does not upload, complete, or delete objects, and it uses saved values, not unsaved edits.

If the connection has no bucket credentials, ingest stays on the replica-safe fallback: the browser asks `/import/start/` for an upload id, then sends `/import/chunk/` pieces of at most 80 MiB (`ISO_UPLOAD_CHUNK_MAX_BYTES`), up to four at a time (`ISO_UPLOAD_CHUNK_CONCURRENCY`). Pieces may arrive out of order. They are stored in PostgreSQL (`IsoUploadChunk`), not on the web container's disk, so the next piece can land on any replica ([11.6](/platform/observability.md#id-116-multi-host-rules)). When every byte is present the last completing piece streams the assembled file to Proxmox (`POST /nodes/{node}/storage/{storage}/upload` with `content=iso`) and deletes the stored pieces.

Dry-run and eager-Celery finalize inline; live production finalizes on a daemon thread in the web process so Cloudflare's 100 s origin-response limit cannot 524 the last origin request. That thread heartbeats `proxmox_sent_bytes` onto the catalog row as the file appears on storage; a write that goes silent is discarded instead of sitting at Importing forever. Library progress is browser-ingest then waiting for the mount (not a 99% cap on received bytes). The browser must stay on the upload page until the file has been sent; in-app navigation asks first, then cancels if they leave (`/import/cancel/`: catalog row, incomplete multipart or leftover object, leftover storage file). Closing or refreshing the tab also cancels an in-progress import. After the file is in the bucket, waiting for Proxmox to list it continues without the browser. The browser aborts a hung piece after 10 minutes on the chunked path and 20 minutes on a direct part. An `importing` row with no new bytes for 30 minutes (`ISO_UPLOAD_STALL_SECONDS`), or a Proxmox write / mount wait that hits `ISO_UPLOAD_TIMEOUT_SECONDS` / `ISO_UPLOAD_APPEAR_SECONDS`, is deleted: the catalog row, the object or stored pieces, and any leftover file on the ISO storage. Refresh from Proxmox does not promote an in-flight `importing` row to `available`. Starting the same filename again replaces a stuck import that is not currently writing to Proxmox. The API token needs `Datastore.AllocateTemplate` (and `Datastore.Audit` to list) on that storage. File bytes and presigned query strings are not written to the audit log. A no-JS form POST of the whole file remains for tiny local uploads; Cloudflare will still 413 anything over its plan cap on the origin path.

Uploads are not kept in this application after import. On the chunked path each request still spills the 80 MiB piece to a request-local temp file (`FILE_UPLOAD_MAX_MEMORY_SIZE` stays small); the durable in-flight copy is the PostgreSQL row, deleted when the import completes or is discarded. `DATA_UPLOAD_MAX_MEMORY_SIZE` still allows the 10 GiB quota for a classic POST that never hits Cloudflare.

**Customer library.** Each organization may store **10 GB** of ISOs (`IsoImage.organization` set; `billing_account` records the uploader and is cleared if that tenant is deleted). Operator catalog rows leave both null. Filenames on storage are prefixed `o{organization_id}-` so two organizations can both upload `ubuntu.iso` (accounts not yet in an organization still use `a{account_id}-`). Every tenant in the organization lists, deploys, reinstalls and deletes the same library. Customers manage it at `/client/services/isos/` (Images): upload a file or delete. Quota counts `importing` and `available` `size_bytes` across the organization. A failed upload zeros `size_bytes` so it does not keep occupying the cap. Clients cannot use operator catalog rows. Delete is refused while the file is mounted on a non-destroyed VM; eject it from the instance page first. An `importing` row can be cancelled from the library; that removes the catalog entry, the stored pieces, and any leftover file on storage, and does not wait for an in-flight Proxmox write to finish.

`IsoImage` is the catalog row: filename, volid (`l1-isos:iso/<file>.iso`), ownership, size, and status (`importing` / `available` / `failed` / `missing`). Writes are gated on `BILLING_PROVISIONING_LIVE`; a dry-run upload records an available row without calling Proxmox. Listing storage content is a read and is not gated.

Filenames must end in `.iso` and contain only letters, numbers, dots, dashes and underscores.

**Deploy and reinstall from ISO.** Checkout offers **Templates**, **Bring your own ISO**, or **Backups** ([5.20](#id-520-backups-and-snapshots)). Reinstall stays Templates or ISO. A cluster with no active templates still offers reinstall when the organization has a ready ISO on that connection. Choosing an `available` ISO from the organization library creates a blank guest (`POST /nodes/{node}/qemu`) with `scsi0` on VM storage, `ide2={volid},media=cdrom`, and `boot=order=ide2;scsi0`. There is no clone, no Cloud-Init password, no in-guest grow-disk or login-policy step. Reinstall still keeps VMID, node and IP (no `release-network`). Switching template ↔ ISO on reinstall clears the other FK. The finished guest is treated like a manual-configuration image: finish the installer in web VNC. After the guest exists, the instance overview (and the operator service page) can attach a library ISO to a free CD-ROM slot or eject it. Attach puts the CD first in the boot order so the next restart boots the installer or rescue image; eject removes that drive and boots from disk. Cloud-Init drives are never overwritten (template guests use `ide3` or another free slot; ISO-created guests reuse `ide2`). The `iso_image` FK is the currently mounted CD-ROM, so library delete is refused until the ISO is ejected. The original install source stays in metadata (`selected_iso_image_id` / `selected_iso_image_name`) so the OS mark and first-run notice survive eject. Local dry-run records the attachment without calling Proxmox.

## 5.20 Backups and snapshots <a href="#id-520-backups-and-snapshots" id="id-520-backups-and-snapshots"></a>

Backup storage is **organization-wide**, not per VM and not per tenant. Customers buy a pool in **10 GB steps at $1.00 per step per month**, billed hourly from account credit on the organization's **payer** (`payment_account_for` / `BACKUP_STORAGE_HOURLY_PER_STEP` / plan `vps-backups`, metadata `attach_to: billing_account`). Quota, cancel, and purge live on `BillingAccountAddons` for that payer. Purchase and upgrade through Images (`/client/services/isos/`) via `set_backup_storage_quota` from any tenant in the org. Super-admin can do the same on the client billing tab. The first hour is charged on purchase only, not on upgrade. Upgrade is increase-only. There is no downgrade except full cancel.

**Cancel** cannot be immediate. `schedule_backup_storage_cancellation` sets `BillingAccountAddons.backup_storage_cancel_at` to **00:00 UTC on the next 1st** (same month boundary as Extra Bandwidth). Billing and use continue until then. `process_due_backup_storage_cancellations` (from `_run_billing_maintenance`, after usage aggregation) zeros the payer quota, cancels the addon `Subscription` on the payer, and starts a **7-day grace** (`BACKUP_STORAGE_GRACE_DAYS`) before `purge_lapsed_backup_storage` deletes **every** snapshot and backup in the organization. Undo before the 1st with `undo_backup_storage_cancellation`. Buying or increasing the quota withdraws a scheduled cancel and a pending purge.

QEMU snapshots and vzdump backups **share the pool**. Used bytes are `max(size_bytes, reserved_bytes)` for non-deleted backups and non-gone snapshots on **every tenant** in the organization. Create jobs take the reserve after locking the organization, then the requesting tenant and payer in pk order, then the VM (`select_for_update` of `("self",)`). A failed create releases the reserve.

A snapshot reserves 1 GiB plus RAM if included. A **backup reserves what the guest is measured to hold**, times `BACKUP_RESERVE_HEADROOM` (1.25) and capped at the provisioned disk. **Only an unmeasurable guest reserves its whole `disk_gb`**, which is what every backup used to do: a 60 GB disk holding 4 GB reserved 60 GB and was refused on a pool with room for ten such archives.

`measure_backup_bytes` takes the newest finished archive of that guest, else `vm_allocated_bytes`. **Both readings are the hypervisor's own; nothing is asked of the guest.** No `agent/get-fsinfo`, no in-guest filesystem read — a guest with qemu-guest-agent off or uninstalled measures like any other, and the figure is not the customer's to influence.

`vm_allocated_bytes` lists the guest's volumes (`/storage/{id}/content?content=images&vmid=N`) and then reads **each volume's own endpoint**, `/storage/{id}/content/{volid}`, summing `used`. The extra call per volume is the whole point: **the listing cannot answer this on ZFS.** `ZFSPoolPlugin::list_images` reads `volsize`, the provisioned size — the very number this replaces — while only `volume_size_info`, behind the per-volume endpoint, reads `usedbydataset`. Any change that "optimizes away" the per-volume call silently restores the old whole-disk reserve on ZFS.

Two properties of that figure: it is the **post-compression** on-disk size, so a compressed pool reports less than the logical data vzdump reads and re-compresses, and blocks the guest freed but never trimmed still count as allocated. The first reads low and the second high; the headroom covers the gap rather than either being corrected for.

The measurement is taken **before** `request_vm_backup` opens its transaction, because it is an HTTP call and those locks are the organization's. `backup_reserve_bytes` itself never calls out — the caller passes what it measured — and an unreadable storage means unknown, never zero. Undermeasuring cannot overbill (the pool is billed on purchased quota, not on use); it means a pool can end an archive slightly over quota, and the real size read off the storage listing replaces the reserve on success either way.

A **backup** writes its real size on success, read off the storage listing, and `reconcile_backup_storage` (beat `reconcile-backup-storage`, six-hourly) re-reads it, marks vanished archives `missing`, and reports archives no row claims. A **snapshot keeps its reserve as its charged size**: the PVE API exposes no per-snapshot size, so a snapshot that grows past 1 GiB of CoW data is undercharged and nothing can currently detect it. Reconciliation corrects a snapshot that is *gone*, not one that has grown; the cap of 5 per VM is the real bound on that exposure. Destroying a VM marks snapshots `gone` (and clears their reserve) and SET\_NULLs `VmBackup.source_vm`; it does **not** cancel the organization pool.

**Snapshots** live on the guest disk. Manual only, cap **5** per VM, restore only onto that guest, die with the VM. Rollback requires an active pool. Proxmox: `POST /nodes/{node}/qemu/{vmid}/snapshot`, rollback, and DELETE.

**Backups** are `vzdump` (`mode=snapshot`) onto cluster `backup_storage_id`. Blank `backup_storage_id` hides backups on that connection. A manual copy stays until the customer deletes it or the pool's grace window ends; only a schedule's own retention prunes, and only its own copies ([5.20.1](#id-5201-scheduled-backups-and-retention)). They survive VM destroy. Listing and deploy are **tenant-scoped**: Images and checkout **Backups** show only `VmBackup` rows for the current tenant (`account_backups` / `available_backups_for_account`). They are not org-shared like ISOs. Restore in-place: stop, restore `force=1 unique=0`, re-apply the plan's cores/memory and grow the root disk when the archive is smaller (`apply-restored-hardware`), reconfigure network, start, grow the in-guest filesystem. That hardware step exists because `qm restore` rebuilds the guest from the archive's own config: without it a server restored from a pre-upgrade backup came back on the old cores, memory and disk while the record kept billing the new plan. It deliberately does not rewrite Cloud-Init credentials -- the guest now holds the archive's login, so `cloud_init_password` is cleared and the customer is sent to password reset. Only backups of that server (`source_vm` or matching `source_vmid` on the same cluster). Deploy a new VM from checkout **Backups**: new VMID, `unique=1`, skip Cloud-Init password, pin to the backup's cluster, refuse if `plan.disk_gb < backup.disk_gb`, grow the root disk to the plan when the archive is smaller, still apply `ipconfig0`. Reinstall does not offer backup as a source. Restore and deploy both require an **active pool**; an `AVAILABLE` archive is not enough after cancel. A vzdump delete that fails once the quota is already zeroed leaves the row **FAILED** (not restorable) and the provisioning job **FAILED** with the provider error, so operators can retry the delete from the jobs page. The file may still be on cluster storage until that retry succeeds.

Deleting a row with **no archive behind it** -- a blank `volid`, or `missing` after reconciliation looked on the storage listing and did not find it -- takes no provisioning job at all: `request_backup_delete` marks it `deleted`, clears its reserve, and returns `None`. Deletion is about the bytes, and there are none. Queuing a job there could only fail on the absent volid, and `_fail_backup_job` puts a failed *delete* back to **AVAILABLE** -- a library entry offering a restore from an archive that does not exist. `missing` is deletable for the same reason: both backup tables show Delete on one, and it is the only way a customer can clear such a row. The adapter treats an already queued no-volid delete as a no-op that succeeds rather than an error.

Pricing is per 10 GB step per month, metered hourly from the **exact** rate with the fraction four decimals cannot express carried into the next hour (`backup_storage_exact_hourly_rate`, `rate_remainder`), so 730 hours of a $1.00 pool bills $1.00 rather than $0.949. Cancelling stops billing on the 1st and opens a `BACKUP_STORAGE_GRACE_DAYS` window; copies are deleted only once `backup_storage_purge_at` passes, and buying again inside the window calls the purge off. Quota, cancel and purge live on `BillingAccountAddons`, not on `BillingAccount`.

Hourly metering is `aggregate_backup_storage_usage` (`resource_type=backup_storage`) against the payer; the monthly equivalent of the purchased quota is included in `account_monthly_recharge_amount`. Beat `run-scheduled-vm-backups` applies due pool cancellations and purges pools whose grace window has ended. Task waits for create-backup / restore-from-backup are **3600s**, and `_wait_for_task` heartbeats the job on every poll: the stale-lease sweep requeues a RUNNING job after `PROVISIONING_STALE_LOCK_TIMEOUT_SECONDS` (900s), so without the heartbeat any backup past fifteen minutes was re-dispatched while the first worker was still polling. `apps/core/checks.py` fails `manage.py check` if that gap reopens.

A task that fails is reported with **the reason out of its log, not just its exit status** (`_task_failed_message` / `_task_failure_detail`). `exitstatus` is all `/tasks/{upid}/status` returns, and vzdump's is the string `job errors` whatever went wrong, so an operator had to pull the log off the node by hand: one real failure was `zstd ... wrong exit status 70` behind a write error on the backup mount, and none of that reached the job. The log is read **once, and only after the task has already failed**; `INFO:` narration and the closing `TASK ERROR:` echo of the exit status are dropped, and anything going wrong in that read is swallowed so it can never replace the failure already in hand. A `WARNINGS: n` exit is skipped outright -- it is the recoverable start-vm case, matched on that exact wording. Local provisioning stays dry-run (`BILLING_PROVISIONING_LIVE=False`) and invents a volid. Support cannot write the operator purchase/cancel routes (default deny).

## 5.20.1 Scheduled backups and retention <a href="#id-5201-scheduled-backups-and-retention" id="id-5201-scheduled-backups-and-retention"></a>

`VmBackupSchedule` is **one row per server**, set from the server's Backups page (`set_backup_schedule` / `clear_backup_schedule`, `apps/billing/backup_schedules.py`). Daily, weekly (weekday) or monthly (day 1–28), at an hour the customer picks, plus `retain_copies` (1–`BACKUP_SCHEDULE_MAX_COPIES`, 30). Beat `run-scheduled-vm-backups`, already hourly, now runs these *before* pool cancellations — cancelling zeroes a payer's quota, and a schedule due in the same tick would otherwise be refused for a pool the customer holds until midnight.

**The time is wall clock plus an IANA zone, never a UTC hour.** `timezone_name` comes from the browser (`static/js/backup-schedule.js`); empty, or a name this tzdata does not know, means UTC rather than a save that fails or a schedule that throws every tick. Storing a UTC hour instead would walk a customer's 02:00 to 01:00 or 03:00 twice a year without being asked. There is deliberately **no minute field**: the beat is hourly, so a minute would promise precision nothing can honour, and the page says the backup starts *within* that hour.

**Due is "this period has not been served", not "the hour matches."** A late tick, a busy server or an in-flight delete would each silently cost a day under hour-matching; under `period_start` they are caught up on the next tick, an outage self-heals, and no period is ever served twice. Saving a schedule stamps `last_run_at` to the current period start when the *timing* is new or changed, so a schedule starts at its **next** occurrence — otherwise "monthly on the 1st" set on the 20th fires within the hour. Editing only `retain_copies` does not stamp, so a run already due today still happens.

**Retention prunes to `retain_copies - 1` before taking the new copy**, which is what makes a pool sized for N copies actually hold N. A delete is a provisioning job on that same server and a server with a job queued refuses the next one, so a tick that prunes **stops there** and the copy lands on the following tick; the due rule above is what makes that safe rather than a skipped day. **Only a copy there is something to restore from counts against `retain_copies`, and a row holding nothing is always given up before a real archive** (`_prune_queue`). A night where vzdump failed leaves a `FAILED` row with no `volid` and no reserve: counting it made retention delete a real archive to stay under a number made up of empty rows, so a customer who asked for three copies was left with two failures and nothing to restore from. A `PENDING` row still counts — it is being written and holds its reserve. The queue is capped by the table's own surplus, so a lone recent failure stays visible rather than being swept the moment it appears. **Retention only ever deletes `kind=scheduled` copies of that same server** — a manual backup was taken deliberately, usually right before something risky, and a count set for nightly copies is not consent to delete it. Turning a schedule off deletes no copies at all.

`prune_scheduled_backups` returns **provisioning jobs queued**, not rows dealt with, because that is what tells the caller the server is now busy: a row with no archive is retired in place without a job, so any number of those can go in one tick without costing the copy its turn.

A skip (no pool, suspended, host down, another job running) records the reason on `last_error` and leaves `last_run_at` alone. A copy that is *accepted* and then fails on the hypervisor records there too (`record_schedule_backup_failure`, called from `_fail_backup_job`) — without it the panel read "Daily at 02:00, keeping 7 copies" with no error while not one usable copy was being produced. Destroying a server deletes its schedule alongside the snapshot and backup cleanup: the VM row survives as `DESTROYED`, so the schedule's CASCADE never fires on its own and the beat would consider a dead schedule hourly forever. The hypervisor size measurement ([5.20](#id-520-backups-and-snapshots)) is taken **outside** the schedule's transaction and passed in, because the copy takes the organization's row locks and an HTTP round trip inside those would queue every other tenant behind one guest's estimate — this is what `request_vm_backup`'s `MEASURE_NOW` sentinel is for.

***


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.layeronecloud.com/platform/vps/images-and-backups.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.