> For the complete documentation index, see [llms.txt](https://docs.layeronecloud.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.layeronecloud.com/paas/architecture.md).
# Application platform architecture
Push code, we build it, it runs as a microvm.
Push code, we build it, it runs as a microvm. This document covers the model, the boundaries that are already enforced, and the two provider integrations that are deliberately still inert.
## The four nouns
```
source ──▶ Build ──▶ Release ──▶ Deployment
│
config
```
* **Build** turns one source revision into an OCI image, pinned by digest.
* **Release** freezes a build together with the configuration in force at that moment. It is never edited afterwards.
* **Deployment** is one attempt at making a release the thing actually running.
The immutability of `Release` is the load-bearing part. Editing a config var does not mutate what is running — it creates a new release from the same build. Rollback is therefore not a special path: it deploys an earlier release, which still carries the config that code ran with. "It worked yesterday" becomes answerable.
## Isolation from the VPS platform
`Product.provisioning_driver` gains a new `MICROVM` value rather than reusing `PROXMOX`. Every existing query compares against `PROXMOX` exactly (`views/common.py`, `proxmox_adapter.py`, `client_services.py`, `provisioning_runner.py`), so nothing that runs today can pick a platform app up by accident.
Jobs reuse the shared engine. `BUILD_APP`, `DEPLOY_APP` and `DESTROY_APP` are routed by `run_claimed_provisioning_job` to `apps/paas/provisioning.py`, the same way web hosting, mail and node access are. That is worth doing rather than building a second queue: stale-lock recovery, bounded automatic retries, `SKIP LOCKED` claiming and the audit trail are all things the existing engine already gets right.
`CLAIMABLE_JOB_TYPES` is `tuple(ProvisioningJobType.values)`, so new types are claimable on arrival — no allowlist edit needed, and none of the trap described in §5.1 of `PLATFORM.md`.
## What is enforced today
**Tenancy.** `_owned_app` resolves by `public_id` and then checks `user_has_active_tenant_access`, raising `PermissionDenied`. Both read and write paths go through it; the negative case is covered by a test that asserts 403 *and* that no build row was created.
**Customer input never becomes code.**
* `start_command` is stored as a JSON argument vector, not a shell string. The settings form splits on whitespace, so `gunicorn app.wsgi; rm -rf /` becomes five arguments, one of which is `app.wsgi;`. There is no shell in the path to undo that.
* Config keys must match `^[A-Z][A-Z0-9_]{0,63}$` before they can be exported into a guest environment.
* Repository URLs must be `https://`. A `file://`, `git://` or `ssh://` URL would either read the builder's own filesystem or skip transport authentication.
* App slugs match `^[a-z0-9][a-z0-9-]{1,30}[a-z0-9]$` and are globally unique, because they form the default hostname.
**Secrets.** Config values are encrypted at rest without exception, via `apps/core/crypto.py`. Sorting "which of these is sensitive" by eye is not a policy — platform config routinely holds database URLs and third-party keys. `is_secret` only decides whether a value is masked in the interface, and it is set by pattern match on the key. Deploy tokens are stored as a hash and shown once.
**Domains.** A hostname on a LayerOne domain cannot be attached to a tenant app. Custom domains carry a verification token and are not served, nor is a certificate requested, until control is proven.
**Build output is untrusted.** It is whatever the customer's build emits, including whatever their dependencies print. It is rendered escaped, in a scrolling ``, never with `|safe`.
## How a build and a deploy actually run
Both go through `compute.MicroVMHost`, which is a Proxmox node addressed for microvm work. Placement is resolved by `services.microvm_clusters()`, which is `billing.virtualization` narrowed to connections marked for microvms — an application cannot land on KVM hardware because the picker cannot see it.
**A build** (`apps/paas/builder.py`) clones the builder template, writes `/etc/layerone/build.json`, runs `/usr/local/bin/layerone-build`, then **stops the guest and converts it into a Proxmox template**. That template is the release. A build that fails destroys its guest instead.
**Four build strategies, chosen per build.** `BuilderKind` is recorded on the `Build`, not only on the `Application`, and the deploy control offers it:
| Strategy | What it does |
| ------------ | ---------------------------------------------------------------------------- |
| `auto` | Dockerfile if there is one, else Railpack, else buildpacks, else native |
| `buildpack` | Cloud Native Buildpacks, exported to an **OCI layout on disk** — no registry |
| `railpack` | Railway's builder, the successor to Nixpacks |
| `dockerfile` | The repository's own Dockerfile |
Per build rather than per application because swapping builders is what a customer does when one fails to detect their project. That is a decision about this attempt, and storing it on the application would make an earlier build's record lie about how it was produced.
Buildpacks and Railpack both resolve **runtime versions** from the project's own declarations -- `.nvmrc`, `engines.node`, `.python-version`, `go.mod` -- which is the thing the native path cannot do. Because the build guest becomes the release, the runtime they download is baked into that immutable template: release v7 keeps Node 18 forever, even after the builder image moves on. A shared runtime image cannot give that guarantee.
Neither reintroduces a registry. CNB's lifecycle exports to an OCI layout on disk and the image is unpacked into the guest; it never leaves.
**There is no image registry and no image.** The obvious design is the other one — build an OCI image, push it, pull it back at every boot — and it needs a registry to run, credentials to reach it, and a pull per instance start, all to move a filesystem from one place on the cluster to another place on the same cluster. A Proxmox template is already immutable (it cannot be started or modified), already lives on the storage the guests boot from, and rolls back by cloning an older one. The registry was solving a problem created by throwing the build away.
The build sandbox sits on the cluster's ordinary bridge with the firewall on and **not** on any project's private network. A build is the least trustworthy thing the platform runs; it has no route to anyone's database.
**A deploy** (`apps/paas/runtime.py`) linked-clones the release's template once per instance, writes `/etc/layerone/release.json` — environment, command, port — and runs `/usr/local/bin/layerone-run`. Instances are created PENDING; the health prober promotes them, and the routing table only receives instances that passed. That is what makes a bad release fail closed.
Initial startup has a ten-minute grace period from instance creation. A new instance stays pending and receives no traffic while migrations or other startup work run, even after three unsuccessful probes. Its first successful probe makes it eligible for routing. Once an instance has been healthy, three consecutive failed probes still withdraw it from rotation. The deployment's ten-minute deadline remains in force, and a deployment that exhausts startup time fails, retains its captured logs, and requests confirmed guest cleanup. Startup grace is separate from the short timeout used for each individual HTTP probe.
Nothing is pulled, nothing is converted, and no privileged command runs on the node.
## Customer input never becomes an argv
Every program invoked in a guest is a fixed path from this codebase, with no arguments. Repository URLs, branches, credentials and config all arrive as a JSON file written with the guest agent first.
This is not only about shells. PVE parameters of type `array` need repeated form keys, and the obvious encoder — the one the VPS side uses — JSON-encodes a list instead. Confirmed on a 9.1.9 node:
```
--command '["/bin/echo","hello"]' -> Failed to execute child process
"["/bin/echo","hello"]"
--command /bin/echo --command hi -> pid 286
```
It does not raise. It runs the wrong program, or none. So `compute.py` owns its own encoding, and a test asserts the repeated-key form.
## The guest images
**One image per connection**, recorded as `paas_template_vmid` on the server connection. The platform never builds it; an operator does, on that node, and the platform refuses to run without it rather than improvising.
It used to be five. Four of those were not carrying their weight. A `runtime` image was never read by anything — an application instance clones the release its own build produced, not a stock image — and `postgres`, `mysql` and `redis` were three hand-built images per node that existed to do what the official upstream containers already do. A build sandbox and a database differ in what they are *told to run*, not in what they are *made of*.
| Must contain | Why |
| --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `udev` | Without it `systemd-networkd` leaves every link `pending`, never applies a `.network` file, and never runs DHCP. The guest comes up with carrier and no address, which looks exactly like a broken bridge. |
| the QEMU guest agent | The only management path. There is no SSH. |
| Docker | Runs the database engines, and buildpacks need it. |
| `pack` and `railpack` | The two builders a customer can choose. |
| `layerone-build` | Reads `/etc/layerone/build.json`, writes `/run/layerone/build.log`. Installs into its own filesystem, because that filesystem becomes the release. |
| `layerone-addon-init` | Reads `/etc/layerone/addon.json` and `docker run`s the image named in it, with the database, user, password and port it was given. |
| `layerone-run` | Reads `/etc/layerone/release.json` at boot and starts the application. Inherited by every release, since a release is a sealed build guest. |
Pre-pull the database images (`postgres:17-alpine`, `mysql:8.4`, `redis:7-alpine` — see `ADDON_DEFAULTS`) into the image so provisioning does not wait on a registry. The tags are pinned: a database that quietly changed major version between one provision and the next is not a managed database.
**No image contains an SSH server.** Not "sshd hardened", not "sshd key-only" — not installed. There is no inbound path to a platform guest and no reason to build one: the platform drives guests through the QEMU guest agent, which is a virtio-serial channel rather than a socket, so a guest can be unreachable from every network and still be fully controllable. An sshd in these images would be an attack surface serving no purpose.
**The runtime image owns the customer-facing console.** A serial getty that drops into a shell *inside the running container*, as the container's user — not a login prompt for the guest. Root on the guest is locked and there is no autologin, so a plain serial console would show a prompt nobody can pass. What a customer wants from a console is `heroku run bash`, which is a shell in their app, and the runtime image is the only thing positioned to give it.
## Firewalling a guest
Every platform guest boots with the PVE firewall enabled, `policy_in=DROP`, `macfilter` and `ipfilter` on, and exactly the inbound rules its job needs. `apps/paas/firewall.py` is the only place that policy is written.
| Guest | Accepts |
| -------------------- | -------------------------------------------------- |
| Application instance | its listen port, from the edge router address only |
| Database | its port, from the project network |
| Build sandbox | nothing at all |
No guest has an SSH rule, because no guest has an SSH server. Rules are applied **before** the guest starts, so there is no window in which it is up and unfiltered, and they are replaced rather than appended on re-apply — PVE rules are positional and renumber on delete, so a replacement removes from the highest position down.
`dhcp=1` is set with the DROP policy. Without it the guest's DHCP lease is dropped and it comes up with no address at all, which looks exactly like a broken image.
Outbound is `ACCEPT` everywhere, and that is a deliberate gap rather than an oversight. Restricting egress is what stops a compromised guest exfiltrating, but doing it properly needs a per-application allowlist, and doing it improperly means a customer's app fails at runtime in a way that looks like our bug.
**These programs are the remaining work.** The orchestration above is written and tested; the images it drives are not built yet, so nothing has run end to end on hardware. The contract is deliberately small — read a JSON file, do one thing, write a result file — so that it can be satisfied by a shell script.
## The console
xterm.js over Proxmox's `termproxy`, not VNC. A microvm boots the kernel directly and runs `vga: serial0` — there is no framebuffer for VNC to show, and the one thing VNC buys on an ordinary VM (firmware, a boot menu) does not exist here. A terminal also gives the customer copy and paste.
`apps/paas/console.py` and `apps/paas/console_asgi.py` mirror `apps/billing/node_shell.py` almost line for line, because that implementation is proven and the differences are small: `/nodes/{node}/qemu/{vmid}/termproxy` instead of the node endpoint, and a tenant check instead of a super-admin one.
Three properties the relay exists to hold:
* **The browser never sees a hypervisor ticket.** `user:ticket` is sent from the relay as the first upstream frame and Proxmox's `OK` is swallowed.
* **The browser cannot send that frame either.** Client input is queued during ticket fetch and upstream authentication and only pumped afterwards, so the first thing Proxmox reads is always ours. The queue is the guarantee.
* **One terminal per session.** `claim_session` is a conditional update, so a leaked token cannot attach a second shell sharing the guest's input.
The session is minted by POST, lives five minutes, and belongs to the one person who created it — not to the tenant. The token travels in a URL, and a URL pasted into a shared channel must not become a shell for everyone who can read it.
The wire protocol was read out of PVE's own client rather than remembered: `0::` for input, `1:::` to resize (the trailing colon is part of it), `2` as a keepalive. The length is a **byte** count; sending a character count desynchronises the whole stream the first time anyone types a non-ASCII key.
## Managed databases
A database is a microvm on the project's private network, full-cloned from the shared guest image. The platform generates its credentials. PostgreSQL, MySQL and Redis run in the pinned upstream containers already present in that image; database data lives on the database guest's persistent disk.
Attaching a database does not inject variables automatically. Each application explicitly maps a connection URL, host, port, username, password or database name to the environment variable names it expects. Mapped values are derived from the current addon row when deploying; they are not stored as duplicate `AppConfigVar` values. Mapped names are reserved against ordinary variables and other database mappings. An addon that is not ready contributes no values.
The database's Variables section also lists the individual connection values for manual setup. Passwords and URLs require an ownership-checked, CSRF-protected reveal and are excluded from session recordings. Manual copies must be updated by the customer when the connection changes.
### SSL by default
The versioned `agents/paas-guest/bin/layerone-addon-init` configures every new managed database to require TLS for network connections. The primary URL always uses SSL, including after a customer opts into plaintext:
| Engine | Primary URL | Default network policy |
| ------------- | -------------------------------------------- | ------------------------------------------------------------- |
| PostgreSQL 17 | `postgres://…:5432/database?sslmode=require` | TLS on 5432; `hostssl` rules require password authentication. |
| MySQL 8.4 | `mysql://…:3306/database?ssl_mode=REQUIRED` | TLS on 3306; `require_secure_transport=ON`. |
| Redis 7 | `rediss://…:6380/0` | TLS on 6380; plaintext listener 6379 disabled. |
The customer must check **Allow non-SSL connections**, save, and then select **Redeploy database**. Saving changes only `Addon.allow_non_ssl`; it does not queue a job or change a running listener. The queued job snapshots the saved choice. `Addon.non_ssl_active` records the last successfully applied policy. Pending changes are shown separately from that policy, and another policy edit or redeploy is refused while provisioning is in progress.
Opting in permits plaintext on PostgreSQL/MySQL's usual ports and adds Redis's 6379 listener alongside TLS 6380. A separate non-SSL URL is available after the policy has been applied and credentials are revealed. Disabling plaintext also requires saving and redeploying. Existing databases need an explicit database redeploy to install the shared TLS tooling; changing portal code or the golden image alone does not alter an existing guest's listener.
### Certificates and verification
Each database generates its own private CA and server key after cloning. No CA key, server key, test database or test credentials are baked into the image. Private keys remain on the database guest. Only the public CA certificate is stored in `Addon.tls_ca_certificate`, and its download checks tenant ownership and returns a non-cacheable response.
The default PostgreSQL/MySQL URLs require encryption without verifying server identity. Customers can download the CA and use PostgreSQL `verify-full` with `sslrootcert`, or MySQL `VERIFY_IDENTITY` with `ssl-ca`. Redis clients using `rediss://` must be configured to trust the downloaded CA. Certificates include the guest's network addresses and localhost as subject alternative names.
The CA lasts ten years and remains stable across redeployments. Server certificates last 90 days. A daily systemd timer renews them when fewer than 30 days remain or the guest's addresses change, restarting the database when necessary. Renewal reads the last successfully deployed guest configuration, so a failed redeploy cannot silently activate a pending non-SSL option. A missing partial CA or a CA approaching expiry fails visibly rather than silently changing the trust anchor.
### Redeployment and image checks
Database redeploy uses the same VMID and verifies its recorded identity against Proxmox before writing to the guest. It upgrades the shared initializer and certificate renewal units through the guest agent, then reconfigures the database container using its exact existing data mount. An unknown data mount or changed engine image tag is refused. It never clones a replacement disk.
The old container is retained, stopped with automatic restart disabled, until the replacement passes authenticated TLS and plaintext-policy checks. Failure restores the previous container and configuration; success removes only the old container, without deleting its data volume. Retrying a failed redeploy uses the same path. Only a failed initial guest whose destruction was already confirmed may return to fresh provisioning.
Readiness requires an authenticated, CA-verified database query and a TLS handshake through Docker's published port. The portal publishes the new connection details and active policy after those checks and firewall updates succeed. Database firewalls limit both permitted listeners to the project network. Connection-policy saves and database redeploy requests are audited without recording credentials.
`scripts/build_paas_guest_image.sh` installs Python, OpenSSL, the initializer and renewal units, then runs the initializer's `--self-test` before sealing. This exercises all three pinned engines, rejects incorrect passwords and plaintext by default, enables/disables plaintext through redeployment, and checks that stored data and the CA survive. A failure prevents sealing. The test containers, data and keys are removed afterwards. These engine tests run on the Proxmox image build; local Django tests exercise policy, ownership, certificates and rollback with a simulated container backend.
### Rebaking an installed microvm image
On a node with an existing `microvm-paas` image, run `scripts/rebake_paas_guest_image.py --source --target --node ` from a bundle containing that script and the versioned `agents/paas-guest` files. This full-clones the existing image, preserving its custom boot tooling, builders and pinned engine images. The source template is never modified.
The target VMID must be unused across the cluster. An interrupted build can be resumed only for the matching unpublished candidate, after its Proxmox operation has finished. Returned task IDs are polled to completion; local `pvesh` progress output is handled separately from the task result. The script checks the inherited guest tools, constrains systemd-networkd to the virtio NIC so it does not capture Docker veth links, and uses the NIC MAC for DHCP identity.
The candidate runs the complete engine self-test before it is stopped and converted to a template. Test data, credentials and private keys are removed, and machine identity is cleared before sealing. Only a verified template should be selected as `paas_template_vmid` on the compute connection. Keep the previous image for rollback and existing release dependencies. Selecting the new image affects future builds and newly provisioned databases; existing databases require an explicit redeploy.
## Importing a repository's `.env.example`
`apps/paas/envfile.py` reads two dialects through one line parser. A pasted `.env` is **values**: strict, and an unreadable line rejects the whole paste. A repository's `.env.example` is **questions**: lenient, because one odd line in someone's example file should not cost them the other thirty, and its comments become the description of the variable below them.
Placeholder values (`changeme`, empty, and connection strings pointing at `localhost`) are offered as prompts rather than defaults — importing an example file should not be a way to deploy with `localhost` in your config.
Only for GitHub-connected applications. Fetching an arbitrary URL from the web tier on a customer's say-so is a request-forgery primitive, and a public Git URL is cloned by the build sandbox instead.
## How traffic reaches an application
Nothing here holds a public address — not an application, not the hypervisor. **Cloudflare is the proxy and the load balancer**; there is no reverse proxy of ours in the path.
```
customer.com ──(custom hostname)──┐
.apps.layeronecloud.com ────┤
▼
Cloudflare (TLS, proxy, load balancer)
│ tunnel, outbound only
▼
cloudflared connector
│ one ingress rule per hostname
▼
application microvm on the project VNet
```
The connector exposes the project networks and does nothing else. It dials out, so no host in this path needs an inbound firewall rule.
**Applications had a dedicated IPv4 and no longer do.** The original reasoning was sound — a stable address is what lets a customer move a workload here without re-doing every partner allowlist — but it charged every application $3/month for something most of them never needed, and reachability was never what an address bought. What it did buy was source-IP identity for an allowlist, which is an *egress* property; if that returns as a requirement it belongs on the egress NAT and should be priced as the add-on it is.
## The ingress table
`apps/paas/routing.py` computes the table and `edge.publish_ingress` writes it. Three properties, none of them optional:
* **Written whole, never patched.** Cloudflare replaces a tunnel's configuration wholesale, so "add one rule" does not exist. Rebuilding from the database every time makes that a feature: two concurrent deploys each write a complete, self-consistent table, and whichever lands second is simply the more recent truth.
* **The last rule is a catch-all with no hostname**, and no earlier rule may omit one. Cloudflare requires the first and would be swallowed by the second. Both are checked before the write, because a malformed list does not partially apply — it leaves the previous table in place and every application silently keeps routing wherever it used to.
* **An application with nothing healthy gets a rule answering 503**, rather than no rule. Without it the request falls through to the catch-all and a customer mid-deploy sees "not found" for an application that exists, which reads as data loss rather than as a deploy in progress.
`edge.sync()` is the single entry point, called on create, deploy, health transition and destroy. One function because the answer is always the same, and because a caller who has to remember which of three calls to make will eventually make two of them. It swallows its errors: a Cloudflare outage must not fail a deploy that otherwise worked, and the next sync converges.
The health prober syncs **only on a transition**. It runs every fifteen seconds, and rewriting an unchanged table that often would be a great many writes to say nothing.
## What this costs
**One instance per hostname.** A tunnel ingress rule maps a hostname to a single service, so an application scaled past one instance does not round-robin — the first instance takes all of it. Spreading across instances needs Cloudflare Load Balancer with the tunnel as an origin pool. `instance_count` defaults to 1, so nothing is broken today, but scaling out is not a matter of raising that number.
**Cloudflare is load-bearing for the data path, not just for DNS.** An outage there takes every application down and there is no fallback, because no origin has an address to repoint at. That is the price of no public addresses anywhere, and it is a different risk profile from owning the edge.
**The connector can reach every tenant network.** One process with a route into all of them. It runs no tenant code, holds no tenant credentials and has no inbound listener, which is the mitigation, but it is a concentration of trust and belongs on a host treated as infrastructure rather than as capacity.
## Interface
`/client/apps/` — list, create, then five tabs per app:
| Tab | What it is for |
| -------- | ------------------------------------------------------------- |
| Overview | Live release, source, recent builds |
| Activity | Releases with rollback, deployment history |
| Config | Environment variables, masked when they look like credentials |
| Domains | Default hostname, custom domains with their DNS records |
| Settings | Runtime settings, deploy token, delete |
Deleting an application requires typing its identifier. It destroys instances, so a misplaced click on the wrong tab should not be able to do it.
## Open items
* The guest images. The build, deploy and addon orchestration is written and tested against a fake hypervisor; the three template images and the programs they must contain do not exist yet, so none of it has run end to end on hardware.
* The console's guest end. The LayerOne side is built (below); what it lands on is the runtime image's getty, which does not exist yet. Until then a console opens onto whatever the guest's serial console shows, which on an image with root locked and no autologin is a prompt nobody can pass.
* Multi-instance applications. A tunnel ingress rule maps a hostname to one service, so `instance_count > 1` boots the guests and routes to the first. Spreading load needs Cloudflare Load Balancer with the tunnel as an origin pool, which nothing here sets up yet.
* A reconcile task. `edge.sync()` runs on create, deploy, health transition and destroy, and swallows its errors so an outage cannot fail a deploy. Nothing yet rewrites the table on a timer, so a table left stale by an outage stays stale until the next lifecycle event.
* Domain verification is modelled and displayed but not yet checked; nothing sets `verified_at`.
* Billing: `Application.plan` exists but no subscription is created. Apps are not billed yet.
* Scale-out: `instance_count` is stored and shown but only one instance is deployed.
* Git push triggers: a GitHub connection, webhook receiver and `auto_deploy` flag now exist (`apps/paas/github.py`); builds can also still be started from the interface or the CLI token.
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:
```
GET https://docs.layeronecloud.com/paas/architecture.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.