> For the complete documentation index, see [llms.txt](https://docs.layeronecloud.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.layeronecloud.com/platform/web-hosting/nodes.md).
# Removal, node sudo policy and agent distribution
Removing a hosted domain is a durable dependency workflow, not one isolated job.
## 6.11 Domain and service removal
Removing a hosted domain is a **durable dependency workflow**, not one isolated job. The portal remains the desired-state authority and records the complete affected resource closure **before any node mutation is dispatched**.
For a domain or subdomain it locks the owning billing account, hosting service, domain subtree and related resources in one transaction, then fences every resource lacking durable node-removal proof with **one new generation** and persists these ordered stages:
1. scheduled tasks targeting a domain or runtime application in the subtree;
2. runtime applications;
3. mail aliases;
4. mailboxes;
5. customer DNS records and exact `layerone-mail:` hosted-mail records;
6. mail certificates (identified by mail-domain UUID);
7. mail domains;
8. domain certificates;
9. descendant subdomains, one depth per stage from deepest to shallowest;
10. the requested domain or subdomain.
**Databases are account resources and are deliberately not part of a domain removal.** Failed, pending, applying, deleting and never-provisioned resources are **included**. A resource already marked deleted is skipped **only** when a same-generation successful typed removal job contains the exact `absent` or `removed` result — legacy or locally projected tombstones are re-issued, so a portal status alone can never stand in for node cleanup.
**Only the earliest unfinished stage may be submitted.** Node-owned resources use the normal typed v2 lane and link to their `WebHostingManagementJob`, so the node HTTP exchange, node events, failure stage, error type and bounded summary stay in the Operations job journal. Customer DNS steps link to the durable provider lane with its lease, bounded retry, failure code and exact-record readback. Hosted-mail DNS cleanup **deletes only records bearing the exact LayerOne marker for that domain** and persists its absence receipt on the workflow step; **it never issues a zone-wide delete or removes an unmarked customer record.**
Node-generated mail DNS instructions alone do not require managed-provider cleanup: with no managed receipt, matching configured zone, or recorded managed publication attempt, the step records `not_managed` and allows removal to continue — which also covers externally managed DNS verified by the customer. **Disabled or historically configured zones and failed publication attempts still require cleanup**, because a provider write may have completed without a current receipt. When a configured zone does match, even customer-verified DNS goes through exact marker cleanup; customer verification does not transfer ownership of previously managed records.
A step succeeds only when **both** hold: its successful management job matches the service, operation, resource type, resource UUID **and fenced generation** and contains the operation's exact `absent`/`removed` result for that UUID; **and** the fenced portal resource is at the same generation and has projected to `deleted`, or its local tombstone has already been removed while that exact node receipt is retained. Dispatch and retry reuse this evidence without creating another removal job or advancing the generation. **A missing portal row without an exact removal receipt remains a failure**, not an assumption that remote cleanup succeeded. **Provider DNS never inherits a node resource's absence proof.** A retained newer-generation node intent also prevents reuse of an older step's receipt.
One failed stage prevents all later stages from dispatching. Retry advances only failed resource generations and creates fresh action identities for those steps; successful steps are never repeated. A periodic coordinator resumes pending workflows after an interruption. Submitting Remove again reuses the active workflow instead of advancing generations twice. **An overlapping parent/child workflow is rejected before any resource changes**; only an exact duplicate request for the same target reuses a workflow. Marking every affected resource `deleting` at admission also fences late success from older ensure jobs and prevents new child, certificate, credential or scheduled-task work from attaching to the subtree during cleanup.
**Whole-service removal.** The customer types the primary domain (or the immutable service ID for an unplaced reservation) into the irreversible confirmation. **Admission first rejects any retained remote backup restore point**, because an exact provider-delete/readback contract does not yet exist — that preflight runs before billing or customer data changes. Otherwise billing is settled through the fixed admission cutoff and stops, every mutation lane is fenced, and a durable crash-resumable workflow removes provider-owned DNS and hosted-mail DNS, domain children, databases and the node account **in dependency order**. The service becomes `Deleted` only after the final typed `remove_hosting_account` result proves that exact account ID absent; its audit/workflow tombstone retains immutable diagnostics while live identity fields are released, so the customer can place a fresh hosting order.
Confirmation first creates a durable **read-only `inspect_hosting_account_removal` action**. While that proof is queued or running the service keeps its current lifecycle, customer data, credentials and billing state, and the service lock rejects new management, file-transfer, runtime-delivery, scheduled-task, customer-DNS and credential-rotation intents (including hosted-mail DNS publication); direct work admitted before the workflow may drain but cannot spawn a new follow-on mutation. A failed proof stays actionable and an explicit retry mints a fresh inspection generation. **Only an exact `ready` result for the selected account applies the billing cutoff and removal fence**, and the final `remove_hosting_account` still independently rechecks the backup authority.
**`Cancelled` and `Ended` are commercial lifecycle states, not node-absence proof.** If a v2 account action was admitted, a legacy authority may have crossed the node boundary, or the portal retains observed account materialization (UID/provisioning/activation), the service stays visible, removable, included in billing termination, and **blocks a replacement order** until the exact whole-account deletion workflow succeeds. An exact pre-admission rejection, or a cancellation that never prepared or submitted node authority, retains the prior terminal behaviour and does not block a fresh order.
Whole-service removal also rechecks the closure of previously completed domain workflows, skipping proven deletions while cleaning remaining legacy children or provider records.
## 6.12 Node administrator sudo policy
**This is for trusted Linux node administrators, not hosted customer users.** Unrestricted sudo is host-wide root access and **is not tenant isolation.**
A sudo invocation is admitted only with **all** of: current wheel membership, an applied panel allowlist entry bound to the **same Linux UID**, and the existing sudoers authorization/authentication. **The panel never creates accounts, adds group members or grants NOPASSWD.** System/service identities and hosting usernames/UID ranges cannot be allowed. Removing a group membership takes effect on the next check; removing the panel entry takes effect when its node job actually applies. **Already-root shells are not revoked, and root can change host policy itself.**
Native sudo **approval and audit plugins** add the gate without replacing sudoers or its policy/I/O/audit plugins. Approval covers normal command execution, shells and sudoedit; the audit gate also checks policy-only list/validate acceptance, and audit hooks capture sudoers failures before approval runs. **The guard makes no panel request during sudo.** Missing/unsafe policy and unavailable local audit storage **fail closed for nonroot** while root retains recovery. A denial claims it was logged only **after** the event has been durably stored.
Sudo 1.9.4+ (plugin API 1.17+) is required; unsupported versions fail explicitly. The fixed source and minimal ISC-licensed ABI header ship in the public bundle and are compiled by the installer on the node. The plugin lives at `/usr/libexec/layerone/sudo_guard.so`, the allowlist at `/etc/layerone/sudo-policy` (root 0600), and native audit records in the root 0700 `/var/lib/layerone/sudo-events` ring. Ownership, symlinks, file types and permissions are checked. Active-plugin replacement and config activation **retain/restore prior working bytes on verification failure**. Fresh installation prepares support but **never silently activates an empty policy.**
Panel side: the node detail page shows administrator access, desired/applied revisions, immutable shared-job attempts and paginated sudo security events. **Only super admins** can edit trust, queue policies, refresh events or retry; client and Support access is denied. **GET renders stored state and never sends node requests or queues work.** *Applied* means verified node readback, not an HTTP acknowledgement.
`manage_node_access` uses the shared `ProvisioningJob` dispatcher, lease and predecessor machinery, with **no fake billing account or VM**. `WebHostingNodeAccessAction` holds an immutable typed request, UUID and trusted connection snapshot. Ambiguous failures reuse the same UUID; confirmed failed node attempts get a new UUID on an explicit authorized retry. **Pin/identity rotation never replays old-trust requests against a newly trusted endpoint.** Security history cannot be deleted or marked resolved through generic job controls, and the desired monotonic policy revision protects against stale work.
Node-scoped operations use the exact `node_id` envelope and reserved `node:` storage ownership; API admission **and** privileged execution both check the installed ID. Tenant requests and purge cannot address node policy. **There is no arbitrary command, path, environment, sudoers text or package API.**
**Transport pinning.** Pin verification happens during TLS connection setup **before** HTTP headers or the bearer key/body are transmitted (`apps/webhosting/node_tls.py`). Every request to a node with a saved pin uses it, including legacy tenant operations sharing that key. Nodes with no pin retain legacy transport for existing operations **but cannot manage sudo.** A certificate rotation requires independent re-verification.
Upgrade workflow: deploy portal/worker code and migrations and rebuild the public bundle → **keep an independent root console open** → update the node with its installer and record the printed node ID and certificate SHA256 **through that trusted console, not an unverified network response** → save that pair under node security (the security ID is separate from the descriptive `remote_node_id`, so existing customer bindings survive) → queue the names of existing trusted wheel administrators (a fresh policy needs at least one; later empty policies may revoke all listed users) → confirm the job completes and the applied revision/UIDs match → test from a separate admin session **before closing root recovery access**.
Root-console temporary recovery, needing no sudo invocation:
```sh
/usr/bin/python3 -I /opt/layerone/current/run.py disable-sudo-guard
```
This removes only the recognized LayerOne plugin lines, preserving sudoers and the saved allowlist/events. A queued policy or installer rerun may enable the saved guard again. Uninstall disables the guard before deleting its files.
Two file-mode traps already paid for:
* **A build temporary does not keep the mode the installer created it with.** Linking ends by chmodding the output to `0777 & ~umask`, so on a node whose root umask is not 022 the freshly built plugin landed 0700 and the strict readback refused the installer's own file one call after writing it. `sudo_policy.BUILD_MODE` is therefore restored after every compile, a directory the safe walk creates is set to its declared mode instead of the masked one, and a file refused for its metadata names itself and reports the mode it had against the mode expected. (A fake compiler that only writes bytes keeps the caller's mode, which is why the suite could not see this; the test doubles now chmod their output the way a linker does.)
* **`/etc/sudo.conf` is the only file in this stack the distribution owns**, so it is held to sudo's own rule rather than an exact mode: a hardened node carried it at 0640 instead of the packaged 0644 and preparation refused the node's own configuration. `sudo_policy.HOST_OWNED` marks those reads and applies sudo's rule (root-owned regular file, no group/world write, no setuid/setgid), and a rewrite keeps whatever permissions the host chose, so adding two plugin lines cannot loosen a file somebody tightened.
**Observations.** Native records contain only sequence, timestamp, invoking UID/username, decision, fixed reason and policy revision. **Arguments, passwords, environment, I/O and source IPs are never collected.** The node ring and portal retention are 2,048 events; explicit refresh fetches up to 100. **This is bounded sudo auditing, not full SSH monitoring**, and retention can drop old events (jobs retain cursor bounds so gaps can be identified).
**CageFS.** CloudLinux CageFS uses **nosuid** mounts. **Never proxy the real sudo binary outside CageFS to make it work.** A root-owned profile supplies a `sudo` *function* only for new interactive Bash login shells inside CageFS, dropping all arguments/input and invoking the fixed `/usr/libexec/layerone/sudo-deny` proxy. That non-setuid native helper accepts only a root proxy execution with a verified tenant UID and **ALWAYS denies**; there is no command-execution or grant path. (The CloudLinux bsock proxy overwrites `PROXYEXEC_UID/GID` using Unix-socket `SO_PEERCRED` after client environment parsing, which is why a fixed deny-only proxy is acceptable and an environment-trusting general sudo proxy is not.) **This function is a friendly warning/telemetry path, not enforcement:** explicit `/usr/bin/sudo`, clearing the function, existing sessions and other shells may bypass the warning but cannot gain privileges through a deny-only helper. **Nosuid remains the jail boundary.** New SSH sessions are needed to load the profile, so acceptance must test actual jailed sessions as well as unjailed administrator sessions.
Local tests cover real TLS pinning, real SQLite/temporary policy files, native C gate/audit logic through an isolated compiled harness, strict node scope, job identity, permissions, retries and migration/rendering behaviour. **They do not establish production CloudLinux/sudo ABI or CageFS acceptance.** Before rollout, test allowlisted and non-allowlisted wheel users, non-wheel users, tenants (including accidental wheel membership), sudoedit, `sudo -n`, `-l`, `-v`, unavailable policy/audit storage, and root recovery on the real node.
## 6.13 Agent distribution
`agents/web-server/` is the canonical standalone node source (tests, systemd units, configuration example, architecture notes). **Django does not import or execute the privileged node package** — the public download view reads build artifacts only, and the downloaded reconciler runs on the destination CloudLinux 9 node as root.
**Build and publish.** The production Dockerfile runs `python scripts/build_webhosting_agent.py` after copying source into the image. Builds fail when required runtime files or the bootstrap are missing or unsafe. `agent-dist/` is in both `.gitignore` and `.dockerignore`, so stale local artifacts cannot override a build. For non-Docker deployments run the same command before starting web workers and keep the artifact directory read-only to the web process.
The archive **allowlist** is the bootstrap's `FIXED_FILES` set plus Python files beneath `layerone/`: root entry points/version, the example configuration, four named systemd units, the default Apache index and 404 pages, and the exact native sudo guard C/ABI header sources (compiled on the node, not in the Django image). The packer reads that set from `bootstrap.py` rather than listing it again, so a new required asset cannot ship in the installer while missing from the tarball. Tests, documentation, `.env`, Git/IDE files, databases, keys, customer files and **all portal application code** are excluded. **Symlinks are refused, and no endpoint can request an arbitrary source path.**
Public GET/HEAD routes:
* `/downloads/webhosting-agent/install.py` — self-contained bootstrap
* `/downloads/webhosting-agent/manifest.json` — version, immutable filename, SHA-256 digest, compressed size
* `/downloads/webhosting-agent/layerone-agent-.tar.gz` — runtime bundle
Unknown names and unsafe/missing artifacts return 404; writes return 405. Downloads are attachments with `nosniff`. Bootstrap and manifest are `no-store`; digest-addressed archives can be cached immutably. **The archive's filename is checked against its contents before serving.**
**Operator workflow and trust.** Install commands use the site's canonical `PUBLIC_SITE_BASE_URL` for both the bootstrap and subsequent downloads, so a stale `APP_BASE_URL` cannot change the origin of code that will run as root. The admin web-hosting list, registration/edit form and node detail show a copyable install/update command; **no API key, node ID, customer content or authenticated URL enters that command.** It downloads the bootstrap to a private temporary directory before starting isolated Python as root — **it does not pipe a download into a root shell** — and requires HTTP 200 and `text/x-python` before starting Python, so redirects and HTML responses produce a download error naming the received status/content type. Curl failures and empty downloads also error and remove the temporary file. Run it on the CloudLinux 9 hosting node, not the portal server or a hypervisor; the bootstrap rejects unsupported operating systems before downloading the bundle.
The bootstrap requires CloudLinux 9 / Python 3.9+, verified HTTPS, a bounded manifest/archive, matching checksum, and **safe regular-only archive paths** (links, traversal, special files and duplicate/unexpected members are rejected) before invoking the host reconciler. It accepts only the existing domain/listen/direct-mail options — **not credentials or an arbitrary source path** — and sends an explicit `LayerOne-WebHosting-Installer/1.0` user agent for both downloads, reporting which artifact failed with an HTTP status or a fixed TLS/DNS/timeout/transport category and **no raw exception text, URLs or response bodies**. Repeating the command reconciles/updates and retains installed configuration and customer state. First installation is noninteractive and uses the system FQDN/default bind address.
**HTTPS authenticates the distribution host. The checksum detects corruption and mixed releases but is not an independent release signature** — a compromised origin could replace both the bootstrap and the bundle. Use a trusted portal origin and secure the pipeline; future signing would need a separately trusted key. During a rolling deployment a digest URL may temporarily hit an older image that lacks it: **installation fails without executing that bundle** and never falls back to an unchecked or different release.
Publishing this code publicly does **not** select an open-source license; none was added.
## 6.14 The node itself (separate deployable)
Full internals live in `agents/web-server/docs/`. The contract that matters from the portal side:
The node is the **operating-system interface** for a CloudLinux hosting node. The portal is the control plane and the source of truth for customer, plan and lifecycle decisions. **The node holds no business policy and is not a second panel.** Its whole job is to expose typed operations, apply them to the OS, and report what actually happened.
```
Portal ──HTTPS──▶ layerone.api ──▶ inbox.db ──▶ layerone.runner ──▶ the OS
│ │
└────── state.db (read-only) ◀──┘
```
Two processes joined only by two SQLite databases and file ownership: the network-facing `layerone.api` runs unprivileged as `layerone-api` and may write only `inbox.db` and `blobs/in`; `layerone.runner` runs as root on a 15 s timer and an inbox path unit and may write `state.db`, `blobs/out` and the OS. **The API cannot mutate the OS and cannot forge a result — the kernel enforces that through file ownership, not application code.** `state.db` uses a rollback journal rather than WAL on purpose: a WAL reader must write the shared-memory index, which would hand the unprivileged process a way to corrupt job state; the cost is that a root crash mid-commit leaves a hot journal only root can roll back, and until the next pass the API returns a clean 503 rather than reading torn state.
**The registry is the point.** Every operation is declared once, and from that declaration the node derives request validation, result validation, job identity, supersession lineage and the account-purge sweep. Two bug classes are made structurally impossible: `registry.seal()` refuses to start if any family lacks a **purge hook** (deletion used to sweep a hand-maintained 24-entry tuple and silently leaked certificates, DKIM keys, mail DNS gates and MySQL quotas), and a handler **never supplies `resource_id`** — the store stamps it, so a result for the wrong resource cannot be expressed.
`layerone.provider` is the only thing that runs a program: an executable must be named by symbol from a fixed table, arguments are always a list, and `shell=True` does not appear in the package. **There is no endpoint that accepts a path, argv, command, package or systemd unit.** Tenant work runs as the tenant inside CageFS and its LVE, starting under `python3 -I -S` so a file a customer places in their own home cannot be imported by a helper root launched. **Customer-controlled content is not a node-health signal**: a missing path, an odd mode or a symlink fails the operation that needed it and nothing else. A failed operation is a **result**, never an outage — it never disables an action, a family, or the API surface.
**Both halves of the contract are derived.** The portal declares one `Contract` per operation: `sends` is compared against the node's request schema and `reads` against its result schema, both by reading the portal's source rather than a copy of it. The result half was once a hand-written table, and that gap cost three families at once — the portal required a `health_status` the node has never produced, validated scheduled tasks against a retired result, and offered PHP extensions the node rejects; each was a job the node applied successfully and the panel then called failed, with both suites green throughout.
***
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:
```
GET https://docs.layeronecloud.com/platform/web-hosting/nodes.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.