> For the complete documentation index, see [llms.txt](https://docs.layeronecloud.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.layeronecloud.com/node-agent/production-readiness.md).
# Production-readiness audit
material defects despite the existing passing suite.
## Verdict
**Not production-ready for customer deployment.** This review found and fixed material defects despite the existing passing suite. All 68 registered operations now complete at least one simulated job successfully, but several real services are represented by provider doubles. No real CloudLinux node was installed, upgraded, deployed to, or tested during this audit.
This is a substantial, uncommitted worktree change, not a small patch. Review the implementation and tests together before recording or deploying it. The wire operation schemas remain compatible with the checked Pulsar contract.
## Verification performed
```sh
python3 -m unittest discover -s tests -t . -v
python3 -m tests.simulate
git diff --check
```
The consolidated suite ran **608 tests: 606 passed, 2 skipped**. The operation audit observed **68/68 successful operation types** and failure results for **34 operation types**, through the real runner and SQLite job completions. This is operation-surface coverage, not complete Python function/branch coverage or proof that every external command works.
The sandbox cannot listen on TCP, so the slow-client TLS concurrency test was also run separately with an approved local listener and passed. The remaining Linux setgid directory-inheritance test cannot be validated on this macOS host. `git diff --check` passed.
Tests use real SQLite, temporary filesystems, transactions, locks, OpenSSL encryption, and tenant-worker subprocesses where possible. A real local Apache parser accepted document roots containing spaces and quotes. Worker tests exercise binary file round-trips and independent runtime application trees. CloudLinux tools, database catalogs/privilege enforcement, mail services, systemd, and most deployment commands remain simulated. SQL doubles model catalog changes and failures; they do not implement either SQL engine.
## Material defects corrected
| Area | Corrections and regression coverage |
| ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Installer | CloudLinux 9/Python preflight; fixed dependencies and required services; CageFS initialization and PHP Selector setup; safe PostgreSQL initialization; scoped loopback authentication and owner-safe HBA replacement; real administrator-query probes; quota-accounting prerequisite; shared runner/install lock and old-runner drain; repair public sealing keys without rotating private keys; fail incomplete runtime/mail/service readiness; unique release directories and bounded canonical pruning. |
| Ownership | Host-wide hostname and database-name conflicts rejected before mutation; immutable physical names and grant pairs; refusal to adopt untracked databases, roles, or vhosts; partial provisioning retains resource ownership. Cross-account tests assert no unrelated resource/filesystem changes. |
| Account recovery | A durable provisioning observation reserves the UID before user creation; retries reuse it and dependent jobs wait for completion. Existing ready observations survive failed reconfiguration. Deletion verifies live identity and refuses account-ID prefix collisions before purging. SQLite schema v3 migrates existing observations transactionally. |
| Database lifecycle | Correct MySQL underscore escaping in grants; checked revocation/drop failures; PostgreSQL SQL errors stop execution; PUBLIC access removed from managed databases; separate NOLOGIN owner/read groups; creator-default read grants; reassign customer-owned objects before user deletion or privilege downgrade; purge retains retry state; credentials sealed before mutation and reused on retries; browser queries run in explicit PostgreSQL read-only transactions. Revocation limits remain below. |
| Mail/certificates | Mailbox passwords hashed from actual stdin, not a literal hyphen; checked map rebuilds, quota/expunge, service reloads and DKIM cleanup; retained removal metadata across partial failures; unreadable account userdb cannot silently erase global entries; certificate rollback and failed-delete retention; uninstall preserves certificate material. |
| Runtime applications | First-upload directory creation; isolated per-application release/environment trees; release and rollback history retained; Python/Node launchers load secrets; build tasks receive environment; safe startup wrappers; no build output in public job progress. |
| Files and cron | Binary content preserved; final symlinks operated on themselves or rejected; unpredictable upload temp paths; bounded larger directory output; runnable cron shell; literal percent escaping; newline rejection before mutation; crontab changes verified before resource state changes. |
| API/runner/store | Conflicting request-ID replay rejected; atomic admission/capacity; no 256-entry inbox starvation; rollback on failed commit; bounded crash retries; stable root-owned upload snapshots; success-only download publication; bounded concurrent HTTP/TLS handling, body framing and socket deadlines. |
| Provider/configuration | Bounded subprocess output and process-group cleanup; binary-safe transport; exceptions/output cannot leak arbitrary customer content; Apache paths safely quoted; failed removals/reloads retain recoverable metadata. |
Focused regression suites include `test_account_recovery.py`, `test_core_readiness.py`, `test_install_host.py`, `test_resource_ownership.py`, `test_database_failures.py`, `test_mail_failures.py`, `test_simulated_operations.py`, and the expanded existing suites.
## Unresolved release blockers
1. **DNS is not wired to an authoritative server.** `operations/dns.py` writes record fragments and reloads `named`, but there is no managed zone declaration/include, SOA/NS setup, or authoritative-server installation. `bind-utils` supplies clients, not this missing integration. A successful reload does not prove the fragment is served. Pulsar/operator input must establish DNS authority and nameserver configuration before this can be implemented correctly; do not advertise DNS publication as qualified.
2. **CloudLinux isolation and tooling are unqualified.** This repository has still never been run on its target OS. Verify kernel/LVE support, license and repositories, exact command flags/output, CageFS mounts, tenant identities, PHP Selector/LSAPI, Passenger, interpreter paths, SELinux and firewall rules. CloudLinux CP Vendors integration (`cpapi`) may be required for this custom control-panel integration and is currently absent. Installing packages alone does not establish these guarantees. Account removal still treats CageFS removal and LVE destruction as best-effort commands; verify their absent-resource semantics and prove no isolation/limit residue survives a failed command before considering purge fully qualified.
3. **PostgreSQL privilege revocation has unresolved gaps.** `_revoke_postgresql()` revokes explicit object privileges only in `public`. A formerly full-access user can retain explicitly granted writes in a custom schema or EXECUTE on a write-capable SECURITY DEFINER routine. It also does not terminate existing sessions: a session that already selected the owner with `SET ROLE` can retain that identity after membership is revoked. These are code-review findings based on PostgreSQL's documented [ownership reassignment](https://www.postgresql.org/docs/13/sql-reassign-owned.html) and [session role behavior](https://www.postgresql.org/docs/13/sql-set-role.html), not locally executed engine reproductions. Do not rely on grant downgrade or removal as complete access revocation until these cases are fixed and tested on the target engine.
The browser's explicit read-only transaction adds protection against write-capable functions invoked by a SELECT, but does not constrain a customer using their SQL login directly. This distinction follows [PostgreSQL's transaction restrictions](https://www.postgresql.org/docs/13/sql-set-transaction.html).
Verify same-account grants and cross-account denial using fresh connections and existing sessions; browser SELECT-only access; current/future tables created by each writer; all-to-read downgrade; user deletion preserving data; and retryable purge. Existing PostgreSQL resources must be re-ensured to provision the new reader group. Audit unmanaged databases for PUBLIC CONNECT/schema privileges: this change deliberately does not alter unrelated databases. Do not treat emitted SQL assertions as evidence of engine-enforced isolation.
4. **Runtime `running` is not application health.** `_observe()` establishes release/linkage readiness, not a successful HTTP response. A crashing app can satisfy the current observation. Exercise real Python, Django and Node applications through Passenger, including startup failures, secrets, restart, rollback, dependency failure and two applications in one account.
5. **Suspension is narrower than full service suspension.** It locks/expires the Unix login and throttles LVE CPU; it does not establish that static HTTP, Passenger or virtual mail access stops. Pulsar must define the desired suspension effects before the node implements additional OS actions.
6. **Mail readiness overstates some observations.** `credential_verified` currently checks a user lookup, not a password authentication attempt; process/configuration readiness is not an external-delivery test. Verify IMAP and SMTP authentication, inbound/outbound delivery, DKIM, DNS, STARTTLS, relay behavior and enforced quotas with real mail clients and recipients. SMTP TLS is operator-managed and deliberately excluded from automatic setup readiness; `tls_verified` covers IMAP TLS only. Manual delivery drills are not installation or setup-job prerequisites.
7. **Quota enforcement and API trust are prerequisites.** The installer fails missing quota accounting but does not rewrite mount options or arrange required reboots. Test actual byte/inode enforcement. As documented in `architecture.md`, Pulsar does not authenticate the node's self-signed TLS certificate: keep that channel on a trusted private network/VPN, or implement certificate pinning in Pulsar before public-internet use.
## Required CloudLinux acceptance run
Use a disposable, licensed CloudLinux 9 node with no customer data and two synthetic tenants. Record package versions, configuration checks and observed results without retaining secrets or customer contents.
* Run a fresh install, an identical second install, and an upgrade; verify service activation, filesystem ownership, dependency repair and unchanged unrelated configuration. Verify the installer pauses its timer/path triggers, drains an in-flight old runner before changing configuration, and restores previously active triggers after a failed upgrade.
* Resolve the DNS authority and suspension requirements above. Test every registered operation through the actual HTTPS API, not only handlers.
* Exercise every supported runtime/PHP version and all database/mail/file/cron lifecycles, including tenant isolation, failure injection and retries.
* Verify external HTTP, DNS and mail responses, plus actual SQL permissions and disk enforcement. Process exit status and file presence are insufficient.
* Interrupt provisioning and removal between steps, rerun reconciliation, then purge one tenant. Verify its keys, quotas, roles, vhosts and scheduled jobs are gone and the other tenant's resources and data are unchanged.
* Test backup/restore and a controlled upgrade rollback before customer rollout. Schema v3 adds a provisioning state that older runners do not understand; do not blindly start old code against an in-flight upgraded database. Keep secrets and credential-bearing files out of test reports.
## Implementation references checked
The fixes were cross-checked against primary documentation for [Dovecot password input](https://doc.dovecot.org/main/core/config/auth/schemes.html), [PostgreSQL role removal](https://www.postgresql.org/docs/17/role-removal.html), [MySQL grants](https://dev.mysql.com/doc/refman/8.0/en/grant.html), and [Apache configuration quoting](https://httpd.apache.org/docs/2.4/configuring.html). Target-host qualification must also use the installed versions and [CloudLinux command documentation](https://docs.cloudlinux.com/cloudlinuxos/command-line_tools/) and [control-panel integration requirements](https://docs.cloudlinux.com/cloudlinuxos/control_panel_integration/).
No staging, commit, push, live installation, or deployment was performed.
## File manager / PHP follow-up — 2026-09-06
The reported live failures have not been reproduced: no affected node/domain, job error, HTTP status, or Apache log was supplied. Source review found and patched these defects locally; this is not confirmation of live recovery:
* File/runtime workers referenced `/opt/layerone/current/...` inside CageFS, but installation did not expose that release there. The fixed, bounded bundled worker source now crosses the tenant boundary as Python's `-c` argument, still with `-I -S`; customer input and secrets remain on stdin. The non-CageFS `runuser` fallback now terminates its options with `--`.
* Standalone PHP Selector setup lacked `native.conf` reconciliation. The installer now verifies native executable paths, creates missing mapping configuration after LSAPI preparation, and checks CageFS selector setup. Missing native `lsphp` triggers vendor setup and executable readback; conflicting PHP handlers are disabled again after that repair. Existing custom/unsupported required mappings are preserved and refused explicitly, not overwritten. FPM is not installed or started, and per-user PHP selections are not changed.
* Installer vhost readback now uses the renderer's document-root quoting, preventing valid roots containing spaces from being rejected as stale.
The file simulation previously substituted a local worker path, hiding the first defect. It now executes the emitted Python command with real subprocess capture, files, and SQLite; only the unavailable OS identity/CageFS wrapper and local interpreter path are substituted. Regressions cover binary transfers, missing-path recovery, cross-home rejection without state changes, runtime artifact unpacking, and secrets staying off argv.
The full suite and `python3 -m tests.simulate` each ran **623 tests: 621 passed, 2 skipped** (sandbox TCP listener and Linux setgid semantics). The simulation observed **68/68 successful operation types** and failure results for **36**. `git diff --check` passed. PHP execution, CageFS identity/interpreter setup, and the reported live failures remain unverified. No staging, commit, push, host installation, or deployment was performed for this follow-up.
Vendor references: [standalone native PHP configuration](https://docs.cloudlinux.com/cloudlinuxos/cloudlinux_os_components/#native-php-configuration), [CageFS command invocation](https://docs.cloudlinux.com/cloudlinuxos/command-line_tools/#running-command-inside-cagefs), and [util-linux option parsing](https://github.com/util-linux/util-linux/blob/v2.37.4/login-utils/su-common.c#L973-L975).
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:
```
GET https://docs.layeronecloud.com/node-agent/production-readiness.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.