> For the complete documentation index, see [llms.txt](https://docs.layeronecloud.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.layeronecloud.com/node-agent/architecture.md).
# Node agent architecture
The node is the operating-system interface for a LayerOne CloudLinux hosting node.
The node is the operating-system interface for a LayerOne CloudLinux hosting node. Pulsar Operations is the control plane and the source of truth for customer, plan and lifecycle decisions. The node holds no business policy and is not a second panel.
Its whole job is to expose 68 typed operations, apply them to the OS, and report what actually happened.
## Shape
```
Pulsar ──HTTPS──▶ layerone.api ──▶ inbox.db ──▶ layerone.runner ──▶ the OS
│ │
└────── state.db (read-only) ◀──┘
```
Two processes, joined only by two SQLite databases and file ownership.
| | `layerone.api` | `layerone.runner` |
| ------------ | ---------------------- | ---------------------------------- |
| runs as | `layerone-api` | `root` |
| triggered by | requests | a 15s timer and an inbox path unit |
| may write | `inbox.db`, `blobs/in` | `state.db`, `blobs/out`, the OS |
| may read | `state.db` | everything |
The API is network-facing and unprivileged. It validates a request, writes it to its own inbox, and answers polls by reading root's database read-only. It cannot mutate the OS and it cannot forge a result — the kernel enforces that through file ownership, not application code.
`state.db` uses a rollback journal rather than WAL on purpose. A WAL reader must write the shared-memory index, which would hand the unprivileged process a way to corrupt job state. The cost is that a root crash mid-commit leaves a hot journal only root can roll back; until the next pass the API returns a clean 503 rather than reading torn state.
## The registry is the point
Every operation is declared once, in one place:
```python
@operation(
"ensure_mysql_database",
family="mysql-database",
request={"id": HEX32, "name": DB_NAME, ...},
result={"state": Str(choices=("present",)), ...},
)
def ensure_mysql_database(ctx, resource): ...
```
From that declaration we derive request validation, result validation, job identity, supersession lineage, and the account-purge sweep. Two bug classes that reached production in the previous implementation are made structurally impossible rather than merely checked for:
**Purge drift.** `registry.seal()` refuses to start the process if any family lacks a purge hook. Deletion used to sweep a hand-maintained 24-entry tuple of directories and silently leaked certificates, DKIM keys, mail DNS gates and MySQL quotas whenever it fell behind. It is now derived.
**Result authorization.** A handler never supplies `resource_id` — the store stamps it. A result for the wrong resource cannot be expressed, so there is no check to omit. The old code hand-wrote a validator per operation, and at least one shipped without comparing the returned resource against the job's.
## Why SQLite
The previous durable state machine was 276 filesystem path constants, 17 `Store` classes, 9 queue lanes and 421 manual `fsync` sites, with no transaction anywhere. Every multi-step mutation therefore had a crash window, and each one was discovered and patched individually — which is where `tombstone` (608 uses), `generation` (1,172), `readback` (235), `fence` (95) and `compensat` (41) came from.
A transaction removes these partial-state windows inside SQLite. Admission, generation allocation and supersession happen in one `BEGIN IMMEDIATE`. Claiming is one `UPDATE`. Completion is one `UPDATE`. OS commands and configuration changes are not part of that transaction: handlers must retain cleanup metadata across failures and reconcile or restore partially applied changes on retry.
## Error containment
`runner.run_once` invokes every step through `_contained()`. A step may fail without starving its neighbours, and there is exactly one place to get that right. The previous design had seven lanes each with its own loop; an unhandled exception in any of them aborted the pass and silently stopped every other lane from draining. That was fixed twice, 35 minutes apart, because the second containment helper was needed for the path the first one missed.
A failed operation is a *result*, never an outage. It never disables an action, a family, or the API surface.
## Privileged execution
`layerone.provider` is the only thing that runs a program. An executable must be named by symbol from a fixed table, arguments are always a list, and `shell=True` does not appear in the package. There is no endpoint that accepts a path, argv, command, package or systemd unit.
Tenant work runs as the tenant inside CageFS and its LVE, via the two workers in `layerone/tenant/`. They start under `python3 -I -S` so a file a customer places in their own home cannot be imported by a helper root launched.
Customer-controlled content is not a node-health signal. A missing path, an odd mode or a symlink fails the operation that needed it and nothing else.
## TLS
`install.py` first issues a self-signed placeholder for the configured node domain so the API can start before Apache is serving ACME challenges. Once httpd is answering `/.well-known/acme-challenge/` on port 80, the installer requests a Let's Encrypt certificate for that same name. Success copies the lineage into `/etc/layerone/tls` (the unprivileged API cannot read certbot's `live/` tree), writes an HTTPS vhost for the hostname that proxies `/v1/` to the local API, and installs a renew-hook so later issuance restarts `layerone-api`. Failure leaves the placeholder in place and is retried the next time install runs.
HSTS on the control plane (including subdomains) is a browser policy. It does not affect Pulsar's urllib probe, and it does not affect HTTP-01: the authority is not a browser, and the node vhost exempts the challenge path from HTTPS redirects. What HSTS *does* do is refuse a self-signed certificate for a hostname under that parent domain, which is why the node's own name needs a public certificate.
Nodes without a saved certificate pin retain legacy encrypted transport without peer authentication for tenant operations.
The connection is still encrypted, so a passive observer cannot read the API key. What is given up on an unpinned placeholder is authentication of the node: anyone able to intercept the connection can present their own certificate and receive a bearer token that manages every account on that node. That is acceptable on a private network or VPN, and not acceptable across the public internet.
Administrator-security operations require an independently verified SHA256 pin and exact installed node ID. `apps/webhosting/node_tls.py` compares the peer certificate before sending any HTTP headers/body. Once a pin is saved, ALL calls honor it because tenant operations share the same bearer key.
## Explicit node administrator scope
The two security operations have registry `scope="node"` and the exact `{request_id, action, node_id, resource}` envelope. Both API admission and root execution compare the target with immutable installed configuration. The reserved `node:` store owner cannot collide with tenant HEX32 identities or participate in tenant purge. No fake tenant is created. Root applies only a typed monotonic username allowlist (locally validated wheel members and UIDs) or returns a bounded page of sanitized native sudo events. Ordinary tenant operations retain their existing envelope and derived identity requirements. See [§6.12](/platform/web-hosting/nodes.md#id-612-node-administrator-sudo-policy) for native enforcement, trust onboarding, recovery and real-node acceptance requirements.
## Testing
Tests run against a real temporary filesystem and real SQLite, with simulated crashes between steps. Nothing mocks the store, the lock or the transaction.
That is deliberate. The previous suite had 1,206 tests passing in 59 seconds, but 42 of its 53 files patched the filesystem and the store away, so it could not observe a crash window, a lane abort, or a purge that missed a directory — which is every defect that actually shipped.
Handlers write system configuration through `config.sysconf()` rather than absolute paths, so those writes are exercised too. A handler that writes straight to `/etc` cannot be tested without root, so in practice it never is.
## Status
| | |
| ------------ | ------------------------------------------------------------------------- |
| Implemented | 70 registered operations, core, tests; see qualification limits below |
| Done | Pulsar's webhosting path speaks this contract, not the two-generation one |
| Not yet done | never run on a real CloudLinux node |
| Not yet done | DNS publication integration and host-level production qualification |
See [the production-readiness audit](/node-agent/production-readiness.md) for tested fixes, simulation coverage, unresolved release blockers, and the required real-node acceptance checks. A passing provider simulation is not production certification.
## Both halves of the contract are derived
Pulsar declares one `Contract` per operation. `sends` is compared against the node's request schema and `reads` against its result schema, both by reading Pulsar's source rather than a copy of it, and Pulsar holds its own declaration to what its result consumers actually read.
That second half was missing, and the gap cost three families at once. Pulsar began requiring a `health_status` of every runtime observation, which the node has never produced; it went on validating scheduled tasks against the retired `layerone_agent` result; and the PHP selector offered sixteen extensions and a version the node rejects. Each was a job the node applied successfully and the panel then called failed, and both suites were green throughout, because the result half of the contract was a table someone had written out by hand.
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:
```
GET https://docs.layeronecloud.com/node-agent/architecture.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.