> For the complete documentation index, see [llms.txt](https://docs.layeronecloud.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.layeronecloud.com/platform/vps/marketplace.md). # Marketplace, power polling and development hypervisors A marketplace deployment is an ordinary template deployment plus one extra step. ## 5.21 Marketplace: one-click applications A marketplace deployment is an **ordinary template deployment plus one extra step**. Clone, Cloud-Init, network, IP filters, boot, login policy and root growth are identical to a plain OS deploy; `install-marketplace-app` (sort 68 on `DEFAULT_PROVISIONING_STEPS`, 87 on `REINSTALL_PROVISIONING_STEPS`) then *arms* an installer in the finished guest and returns. **It never waits for the install.** `agent/exec` is budgeted at `GUEST_AGENT_EXEC_TIMEOUT_SECONDS` (300) and holds a Celery worker for as long as it runs, so what it executes is a *launcher*: `marketplace_scripts.py` writes a systemd unit and starts it `--no-block`, returning in about a second. The installer then has the unit's full hour, survives the worker moving on, survives a reboot (it is enabled, not just started), and reports progress back over a signed callback. A failure to arm is a **warning, not a failed job** — the server is built, running and billable by then, exactly like `grow-disk`. Every deployment carries this step; it records `skipped_marketplace_install` when no application was selected. **Delivery** (`MarketplaceAppDelivery`): `SCRIPT` is a first-party installer living in the codebase (Easypanel); `RECIPE` is a partner submission compiled to a script; `IMAGE` is a pre-baked template needing nothing run. Only `IMAGE` skips the guest work. **Easypanel** does not use upstream's `curl | sh`. That installer lets `easypanel setup` run `docker swarm init` with no `--advertise-addr`, which fails on any guest holding more than one address — which is what an additional-IP or rotated-primary server is. We run the decomposed Docker path and pin the advertise address to the primary IP. It is Ubuntu-only because upstream's Docker apt source is keyed on `UBUNTU_CODENAME`; `base_templates` enforces that and the installer re-checks `ID=ubuntu` itself. ### The callback is not a trusted input The customer has root on the machine posting it. Everything it writes — phase, message, log tail, generated secrets — is **display state**. It must never gate billing, provisioning authority or access. The signed token (`TimestampSigner`, 8h, nonce rotated on every arm so a reinstall invalidates the previous guest's token) identifies *which* install a report belongs to; it does not make the report true. ### Partner recipes are data, not scripts `marketplace_recipes.py` compiles a declarative recipe — `packages`, `file`, `replace`, `service`, `docker_compose`, `run` — into POSIX `sh`. Three properties carry the security: * **Customer input cannot become code.** `substitute_into_shell` emits a quoted shell expansion, never raw text. * **Paths, package names and service names cannot be templated.** A path built from customer input is a write-anywhere primitive. * **Secrets are generated in the guest** from `/dev/urandom`, alphanumeric only. They are not in the recipe, not in the database at compile time and not in the compiled script — only the instruction to generate one is. They return through the callback and are stored encrypted (`apps/core/crypto.py`); the authoritative copy stays in `/etc/layerone/marketplace-secrets.env`. `run:` is the deliberate escape hatch and is what review exists for. It is flagged on the queue and on the version page. ### Submission and review Invite-only (`MarketplacePartner`), portal at `/marketplace/`, outside `/client/`. The identity is the ordinary email-keyed account — the invitation grants the *role*, not a second sign-in. Review is at `/console/billing/marketplace/`, **super admin only and deliberately absent from the Support allowlist** in `apps/accounts/staff_access.py`. ``` draft ──submit──> pending ──approve──> approved ──publish──> live ^ │ └─request changes──┘ ``` **An approved version is frozen forever.** Editing one forks a new version that reviews from scratch. `approve()` recompiles and compares against the hash recorded at submission and refuses on a mismatch — otherwise an approval could be obtained on one script and a different one published. `CHANGES_REQUESTED` is editable in place because nothing was approved yet. Publication defaults to **unlisted**: deployable by direct link so a reviewer orders it exactly as a customer would and watches it install, listed as a separate deliberate action afterwards. ### Partner sandboxes A sandbox runs code nobody has reviewed yet, as root, on our hardware, with a public address. The containment is the feature: * **No inbound anything.** Inbound policy DROP with no rules, which covers SSH without a rule that mentions port 22. Access is the brokered VNC console, which reaches the guest through Proxmox `vncproxy` and needs no guest port. * **No outbound mail.** 25, 465 and 587 are dropped outbound so a sandbox cannot become a spam relay that rebuilds itself every fifteen minutes. Everything else is open because installers need mirrors and registries. * **One per partner**, enforced by a database constraint rather than a view check, because two tabs pressing Launch is how that rule gets broken. * **It expires.** `billing.expire_marketplace_sandboxes` runs every minute. The browser countdown is display only; the sweep is what destroys it. Partners rebuild whenever they like, which restarts the clock. Sandboxes reuse the ordinary provisioning engine against an internal order that is never invoiced, because a test that skips clone, Cloud-Init, network and firewall has not tested what partners will ship on. They install from an **inactive, unlisted** `MarketplaceApp` row, so no customer-facing surface can reach one. Configure the base image, plan, internal account and lifetime under Marketplace → Settings. ## 5.22 Power state polling **What the portal shows as a server's power state is read back from Proxmox every minute, not inferred from the last job we ran.** `VirtualMachine.status` is a record of *intent*. The job engine writes it when a lifecycle job finishes (`_finish_job_succeeded`), and for years the only thing that ever checked it against a hypervisor was suspension reconciliation ([4.10](/platform/billing/credit-automation.md#id-410-credit-automation-the-000-floor-grace-window-suspension-termination)) -- a full fleet audit once a day, plus a five-minute watchdog covering suspended guests only. Anything that moved a guest out of band left the panel confidently wrong: a node reboot, a kernel panic, an OOM kill, an operator clicking Stop in the Proxmox UI. The customer saw **Running** for up to twenty-four hours and had no way to tell that apart from a healthy server. A status that is always Running is not a status. `apps/billing/power_state.py` owns this. `billing.poll_hypervisor_power_state` runs on beat every `HYPERVISOR_POWER_POLL_INTERVAL_SECONDS` (60). **Observation is stored apart from intent.** `hypervisor_power_state`, `hypervisor_power_lock` and `hypervisor_polled_at` are written *only* by this module and hold only what Proxmox said. `status` keeps its existing meaning and its existing writers. Two columns, because the questions are different: "what did we ask for" and "what is actually true" have to be able to disagree, and that disagreement is the signal. **One request per cluster, not per guest.** `/cluster/resources?type=vm` returns `status` and `lock` for every guest on a cluster in a single call, which is what makes a one-minute cadence affordable at all; the per-guest read the reconciler uses would be one HTTP request per VM per minute. The read is deliberately not gated on `BILLING_PROVISIONING_LIVE` -- same reasoning as `node_metrics`, that flag guards *writes*. **Proxmox has no "starting" or "stopping".** Its `status` is only ever `running` or `stopped` (`unknown` when the node is unreachable), and a power change takes no config lock, so there is nothing to poll for a transition. What it does offer is `lock` (`backup`, `clone`, `migrate`, `rollback`, `snapshot`, `suspending`), `qmpstatus` on the per-guest endpoint, and the cluster task list. **We already own the transition locally**: every power request creates a `VirtualMachineAction`. So Starting/Stopping/Restarting is *derived* by `power_display_for` from the in-flight action row, and Proxmox contributes Running, Stopped, Paused and Busy. Deriving it rather than storing it is load-bearing. `VirtualMachineStatus` is branched on by roughly forty modules -- power gating (`QUEUEABLE_POWER_VM_STATUSES`), runtime metering, backups, the firewall, the client API, campaign audiences. Two new stored values would have to be correctly audited into every one of them, and a server that became un-actionable or unbillable while "Starting" is a worse bug than the one this fixes. **A transition is bounded.** An action older than `POWER_TRANSITION_MAX_SECONDS` (15 minutes, comfortably past `GRACEFUL_SHUTDOWN_TIMEOUT_SECONDS` plus the adapter's task waits) stops counting as a transition and the chip falls through to the observation. A worker that died must not be able to pin a chip on "Starting" forever; that is the same lie as a permanent "Running" in a different word. **The paused-guest trap.** Proxmox reports a RAM-paused guest as `status: running` with `qmpstatus: paused`, and `/cluster/resources` carries no `qmpstatus` at all. Suspension reconciliation repairs a paused guest in *opposite directions* depending on the local row -- `_queue_resume_repair` when it says RUNNING, `_queue_stop_repair` when it says STOPPED. A poller that promoted a bare `running` onto a row saying STOPPED would therefore flip that next decision from "stop it" to "resume it", starting a guest the platform had deliberately stopped. So **every disagreement is confirmed with one per-guest `_read_live_power_state` before `status` is touched**, which also covers `/cluster/resources` being served from pmxcfs and lagging a power change by a few seconds. In the steady state that costs nothing; the confirmations are capped per run (`POWER_CONFIRMATION_LIMIT`) so a node dropping out cannot spend the whole minute, and unconfirmed rows are simply corrected on a later tick. Paused is also picked up for free by the five-minute bandwidth/metrics sweep, which already pays for a `status/current` per guest and is therefore the only fleet-wide read in the platform that has `qmpstatus` in hand (`observe_status_payload`). Guards on the write-back, all required together: the observation is `running` or `stopped`; the row currently says RUNNING or STOPPED; no Proxmox lock is held; no provisioning job is in flight for that VM; the account has no billing hold. The correction itself reuses `_correct_recorded_power_state`, which takes the account lock and re-reads the row -- it records `power_state_reconciled_by` so the daily audit and this poller stay distinguishable. Traps already paid for: * **A cluster that fails to answer changes nothing.** Writing `unknown` on our own inability to reach the API would turn one expired token into a fleet-wide outage on every customer's dashboard. * **A guest the cluster did not mention becomes `unknown`, never `stopped`.** Its node may simply be down. "We cannot see it" and "it is off" call for different responses, and `status` is left alone for the daily audit or a human. * **A row that has never been polled renders from `status` exactly as before.** A fresh install, a dry-run deploy and a cluster that has never answered all look the way they always did, rather than showing an "Unknown" nobody has seen. **Rendering.** `templates/billing/_vm_power_status.html` is the only VM power chip; it carries `data-l1-power-state` and `static/js/vm-power-state.js` repaints it from `client-service-power-states` / `admin-service-power-states` -- one request per page, not per server. It polls at 20s, tightening to 5s while anything on the page is transitional, so a customer who presses Stop watches it resolve. List views call `attach_power_displays` so the in-flight-action lookup stays one query per page. The console's hypervisor inventory ([§5.14](/platform/vps/operator-management.md#id-514-unmanaged-client-vms)) deliberately keeps rendering the raw guest state beside the local record: that page exists to show drift, so it must not be fed the reconciled view. The client API exposes the observation as its own `power_state` field rather than adding values to `status` ([§7](/platform/client-api.md)). *** ## 5.23 Development hypervisors and the Development tenant A Proxmox connection carries a **purpose**: `production`, which is every connection that existed before the column and the default, or `development` -- hardware kept for testing the platform itself. Adding a spare box to the panel must not make it available to paying customers, so a development connection is excluded from every customer-facing image picker and refused by order intake. Exactly one development connection may additionally be marked **the Development hypervisor** (`is_development_default`), enforced by a partial unique index plus a check constraint tying it to the development purpose, not only by the form. Both fields live on the connection editor under **Purpose**, and the Connections directory badges a development row. An active **Super admin** automatically gets a tenant named **Development** inside their own organization, pinned to the selected hypervisor through `BillingAccount.development_cluster`. That column is both the marker and the pin: non-null means "this workspace is development", and the row it names is the only connection its services may be placed on. The pin runs both ways, and that symmetry is the whole safety property -- a development connection can only carry a tenant pinned to it, and a pinned tenant can only deploy onto the connection it names. `apps/billing/development.py` holds the rule in two functions and nowhere else. `clusters_for_account` narrows a connection queryset; `development_placement_blocker` answers whether one connection may carry a service for one tenant. Both **fail closed**: passing no account reads as "a customer", so a surface nobody remembered to update hides development hardware rather than offering it. `proxmox_templates_for_plan` takes the tenant and is scoped through the first; order intake asks the second, which is what makes a stale deploy page, the client API and the console's create-a-service path give the same answer as the picker they bypassed. The fallback connection for a first ISO import and a first private network is scoped the same way, because "any enabled connection" is exactly the branch a brand-new customer takes. A development tenant ignores the product's pinned connection, which steers production placement only; honouring it would leave every pinned plan undeployable on the one hypervisor that tenant exists for. `ensure_development_tenant` is idempotent and never raises. It is the one path allowed to create a billing account for a member of staff, deliberately: staff normally have none, so there is no organization to hold a tenant, and the daily super-admin credit reset ([4.18](/platform/billing/revenue-and-reports.md#id-418-daily-super-admin-account-credit)) already assumes the account exists once it does. What keeps it from minting an account for every staff login is the pair of checks in front of the creation -- Super admin exactly (Support and Ticketing only provision nothing), and a Development hypervisor actually selected. It runs from sign-in (`accounts.services._anchor_organization`) and from the two writers of somebody's role, never from a page render, because a client page GET is required to write no billing state ([1.10](/platform/identity/organizations.md#id-110-organizations-and-tenants)). Three deliberate refusals. The pin is **not** re-pointed when the selected Development hypervisor changes: the tenant's guests live on the connection it names, and moving the pin would strand them on hardware the tenant may no longer touch. The Development tenant **cannot be deleted** from the Tenants page -- sign-in would only put it back. And a super admin who already has a tenant named Development that the platform did not create keeps it untouched, rather than having somebody's existing servers pinned to development hardware. The tenant shares the organization's credit pool like any other, so the daily $100 reset still happens **once per organization**, not once per tenant. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.layeronecloud.com/platform/vps/marketplace.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.