> For the complete documentation index, see [llms.txt](https://docs.layeronecloud.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.layeronecloud.com/platform/setup.md). # LayerOne Source project: `LayerOne` (migrated from `README.md`). Run commands from the code checkout and directory specified below, not from this documentation repository. > Internal developer handbook for LayerOne LLC. This is a private production repository: do not copy customer data, credentials, provider tokens, private operational details, or console screenshots into public issues or services. LayerOne is the production platform for LayerOne's VPS hosting business. A single Django codebase serves three primary audiences: 1. **Public marketing site** — homepage, pricing, features, blog, contact, looking glass, status pages, and CMS-managed content at the site root. 2. **Client portal** (`/client/...`) — billing, VPS orders, services, account credit, payment methods, floating IPs, API access, and browser VNC console. 3. **Operations console** (`/console/...`) — internal ticketing, incidents, monitoring, inventory, audit, billing administration, and website CMS. Client ticketing intentionally lives at `/console/tickets/`. The platform runs on Railway with Django/ASGI, PostgreSQL, Redis, Celery Worker, and Celery Beat. Local development uses SQLite, an in-memory cache, eager Celery tasks, and console email delivery. ## Start Here Read documentation in this order before changing a subsystem: 1. [`AGENTS.md`](https://github.com/LayerOne-LLC/LayerOne/blob/main/AGENTS.md) is the short always-on AI brief: global rules, security boundaries, and which documentation page to read. (`CLAUDE.md` is a pointer to it.) 2. The **developer docs** at are the platform reference: what every feature is and how it is intended to behave. [`/ai/agent-guide`](https://docs.layeronecloud.com/ai/agent-guide) routes a task to one page. 3. This README covers setup, architecture, development workflow, operations, and the current product shape. When implementation and the documentation disagree, investigate before changing either one. Do not silently rewrite established behavior. ## Repository Map ``` apps/ accounts/ Authentication, access modes, MFA, passkeys, activity api/ Client API, API keys, developer UI, request metering billing/ Catalog, wallet, orders, Stripe, provisioning, IPAM, VMs webhosting/ Shared-hosting catalog, LET Beta, nodes, jobs, controls core/ Dashboards, health, email, agent API, release tasks marketing/ Public site, CMS pages, blog, looking glass sites/ Multi-site configuration and template context status_pages/ Public status-page data model tickets/ Client and operations ticketing incidents/ Incident lifecycle and publication infrastructure/ Inventory and topology monitoring/ Checks, uptime, and remote execution audit/ Append-only operations audit trail campaigns/ Lifecycle and marketing email campaigns recordings/ First-party public/client session replay config/ settings/ Shared, local, and production Django settings urls.py Root route composition scripts/ Smoke, navigation-audit, startup, and asset-build scripts static/ Console design system and compiled/static assets templates/ Public, client, auth, and operations-console templates LayerOne Agents/ Standalone monitoring agent and systemd installer ``` ## Local Development ### Prerequisites * Python 3.12. * Git. * No local PostgreSQL, Redis, Stripe, or Proxmox service is required for the default development path. `manage.py` defaults to `config.settings.local`. That configuration uses SQLite, LocMem cache, eager Celery, console email, and dry-run provisioning. The checked-in defaults are sufficient for tests; create a local `.env` only when you need explicit integration overrides, and never commit it. ### Windows PowerShell ```powershell py -3.12 -m venv .venv .\.venv\Scripts\python.exe -m pip install --upgrade pip .\.venv\Scripts\python.exe -m pip install -r requirements.txt .\.venv\Scripts\python.exe manage.py migrate .\.venv\Scripts\python.exe manage.py seed_initial_site .\.venv\Scripts\python.exe manage.py runserver ``` ### macOS and Linux ```bash python3.12 -m venv .venv .venv/bin/python -m pip install --upgrade pip .venv/bin/python -m pip install -r requirements.txt .venv/bin/python manage.py migrate .venv/bin/python manage.py seed_initial_site .venv/bin/python manage.py runserver ``` For disposable local logins, run `manage.py create_debug_accounts`. It creates four development-only accounts and fills the three client ones with sample data (servers, credit, invoices, bandwidth, a support ticket), so the portal renders against something other than empty lists: | Username | Password | What it is | | -------------- | -------------- | ------------------------------------------------------ | | `admin` | `admin` | Super admin staff; lands on the operations console | | `user` | `user` | Personal client account with two servers | | `organization` | `organization` | Client account owning an organization with two tenants | | `orguser` | `orguser` | Member of that organization; no billing access | They are listed on the sign-in page while `DEBUG=True`, and only the ones that have actually been seeded appear there. `manage.py create_debug_admin` still creates just the first. All three refuse to be created or to authenticate when `DEBUG=False`, on every credential path including Django admin's own login form. Re-running is safe: it resets the passwords and leaves existing sample data alone. ### Application URLs These are the canonical production routes. A local development server exposes the same paths on its configured development origin. **Public marketing** * Homepage: * Pricing: * Blog: * Looking glass: * Public status page: * LayerOne public status: * Public status API: * Web-hosting pricing: * LET Beta: **Client portal** * Web-hosting services: **Operations console** * Sign in: * Console home: * Admin dashboard: * Client tickets: * Admin ticket queue: * Incidents: * Inventory: * Monitoring: * Audit log: * Billing admin: * Website CMS: **Client portal** * Client home: * Services: * Billing and credit: **Health** * Health check: * Liveness: * Readiness: ## Development Workflow 1. Read the owning app and the matching documentation page before editing. 2. Check the working tree and preserve unrelated in-flight changes. 3. Follow the targeted QC rules below. Security-sensitive changes need a negative test proving that forbidden actions do not change state. 4. Generate migrations with Django when a model or `TextChoices` value changes; do not hand-write migrations. 5. Run the narrowest required feature tests; do not run the full suite or smoke tests unless explicitly requested. 6. Review the final diff for secrets, unsafe rendered content, and accidental generated or unrelated changes. Do not commit unless explicitly requested. ### Required QC After Code Changes Copy, labels, CSS, docs, and small local edits do not need application tests. For major features or billing/auth/provisioning/API behavior changes, run the narrowest relevant test label. The runner already configures parallelism, hashing, and its migration template; pass no flags. ```sh .venv/bin/python manage.py test ``` On Windows, use `.\.venv\Scripts\python.exe`. If a model field changes, also run `manage.py makemigrations --check --dry-run`. Do not run the full suite or `scripts/smoke_all_pages.py` unless explicitly requested. See [Testing and QC](/platform/testing.md) for the suite cap and behavior-test requirements. ### Non-Negotiable Invariants * Users are keyed by email. Passwordless users created by public contact flows are claimable placeholders, not full accounts. * Console views require the repository's `admin_required` access-level check. Every client-facing mutation must independently verify resource ownership. * Sign-in failures remain generic. Passkeys are a second factor after a password, never a passwordless sign-in path. * Provider secrets are write-only in forms. Recoverable secrets use the Fernet helpers; encrypted database columns are never read or written directly. * Stripe webhooks verify signatures and remain idempotent. Paid-invoice wallet credit goes through `record_account_credit_for_paid_invoice` exactly once. * User content is never rendered with `|safe`. Only explicitly admin-authored CMS HTML may use it. * Public canonical URLs intentionally use `https://layeronecloud.com`. * Console design tokens live in `static/css/tokens.css`; shared components live in `static/css/components.css`. Keep duplicated sidebar behavior synchronized. See [`AGENTS.md`](https://github.com/LayerOne-LLC/LayerOne/blob/main/AGENTS.md) for the always-on constraints, and the matching page of the [developer docs](https://docs.layeronecloud.com) for subsystem invariants. ## Billing Flow LayerOne billing is account-credit based. The billing account is the wallet, the account ledger is the source of truth for the wallet balance, and paid invoices are receipts for payments that actually succeeded. Unpaid checkout attempts are internal payment artifacts only: they do not have due dates, they are not shown as customer invoices, and they must not trigger dunning, past-due status, suspension, or service deletion. ### Core Records * `BillingAccount` tracks the customer wallet, billing status, and current suspension state. * `AccountLedgerEntry` is the balance source of truth. Credit-like entries increase balance; usage/debit entries decrease balance. At most one credit entry may exist per paid invoice (enforced by a partial unique constraint). * `Order` represents a VPS purchase or account-credit top-up. * `Invoice` represents the payment artifact tied to an order. Open invoices have `due_at=None` and stay hidden from receipt views. Paid invoices are the customer/admin payment receipts. * `PaymentAttempt` records the provider request and idempotency state for a checkout, saved-method charge, auto-recharge, or webhook reconciliation. * `Subscription` links the purchased service to its current commercial state. * `VirtualMachine`, `FloatingIPAllocation`, and usage records describe the billable resources that produce runtime ledger entries. Expected result: abandoned checkout attempts leave no customer-visible debt. Only successful payments create visible receipts and add credit to the account ledger. ### Order Intake And Payment 1. A customer picks a VPS plan, billing cadence, and template. 2. The order form creates an `Order` for the selected plan and cadence. 3. Monthly and annual orders require the upfront plan price. 4. Hourly orders may require a one-month reserve unless the account already has eligible credit, a default payment method, a previous hourly deposit, or an active subscription. 5. Checkout creates an open invoice and payment attempt with no due date. 6. A successful provider confirmation marks the attempt succeeded, marks the invoice paid, marks the order paid, and posts account credit for the paid amount. 7. A paid VPS order queues provisioning. A paid credit top-up returns the user to the billing page. Expected result: the customer either has a paid order with account credit and a queued provisioning job, or they have an incomplete checkout artifact that does not affect account status. ### Payment Providers Stripe provider settings are managed in the console. Provider secrets are write-only in forms and must not be rendered back to the UI. Stripe checkout is the live order-payment path. Confirmation is idempotent and can enter a verifying state when the provider returns an ambiguous response, so the receipt page can retry verification without double-charging. PayPal supports one-time account-credit purchases and customer-approved monthly subscriptions that deposit account credit. Verified provider confirmations post credit through the same paid-invoice ledger path. PayPal Reference Transactions and saved-agreement charges remain disabled; historical records and the legacy framework are retained. Existing legacy customer-linked PayPal payment methods can be retired with: ``` python manage.py revoke_paypal_payment_methods --apply ``` See [PayPal credit setup and acceptance](https://docs.layeronecloud.com/platform/billing/payments#id-45-paypal) for the REST app configuration, webhook events, monthly cancellation, and payment recovery. Stripe payment-method setup and webhook reconciliation are supported for saved payment methods and automated charges. Hosted Stripe checkout for live order payment currently returns a gateway-not-enabled state instead of collecting money. Expected result: provider callbacks and receipt-page retries can be repeated safely. A payment should post credit exactly once. ### Account Credit Customers can add credit from the client billing page. Top-ups create a planless order, then follow the same provider attempt and paid-receipt flow. The visible balance is calculated from ledger entries, not from invoice totals. Runtime charges are recorded as usage/debit ledger entries with a balance-after snapshot for display. Expected result: the client billing page shows the current credit balance, paid payment receipts, a selectable monthly daily-usage graph, and per-resource usage totals for the selected month. ### Provisioning Paid VPS orders and eligible web-hosting reservations use the shared billing provisioning job engine. VPS jobs are idempotent by order and service type. A web-hosting reservation creates a first-class `ProvisioningJob` for its `WebHostingService`; older reservations missing that link are backfilled when they are placed or reconciled. The operations console Catalog page has independent **VPS hosting in stock** and **Web hosting in stock** switches. Turning one off blocks only new order intake for that product (including VPS API deployments). Paid VPS orders, existing servers and hosting services, queued provisioning, reinstall, and redeploy actions remain available and never consult the stock switches. The shared dispatcher claims both VM and web-hosting jobs and routes each one to its provider adapter. Proxmox jobs retain their ordered VM steps and dry-run behavior. A reinstall retry that is more than 30 minutes old completes as passed with its unfinished steps skipped, so delayed destructive work cannot wipe a server. VM actions requested while another job owns the server are stored as dependent jobs and claimed in request order instead of being rejected or run concurrently. A web-hosting job selects an enabled online node, submits the node's durable idempotent account job, observes its result, verifies the returned identity, and activates the service. The node-side job remains the authority for its root-only CloudLinux and Apache mutations; the shared portal job owns attempts, logs, retries, and customer-visible provisioning status. Initial deployment failures move directly from running to a delayed pending retry inside the failure transaction, with a bounded automatic retry budget. This keeps dependent jobs behind their predecessor. The Jobs page shows the next attempt and offers **Retry now**, which makes a delayed job immediately due for the next dispatcher pass without running provider I/O in the web request. Repeated web-hosting polling backs off, and the Jobs-page log totals use independent indexed counts so a long-running node job does not make the page progressively slower. Expected result: every VPS deployment or web-hosting reservation has one queued or completed shared provisioning job, and a service is not activated until its provider work has been observed and verified. Once active, each customer change is a separate immutable `WebHostingManagementJob`. Domain/subdomain, reviewed Apache policy, account- scoped PHP Selector, Python/Node application, MySQL/PostgreSQL, hosted-mail, backup, and tenant-file metadata controls use closed schemas. The portal sends no executable, raw httpd configuration, host path, password, environment value, or customer file content in these jobs. Immediate post-commit Celery dispatch polls itself through every node in-flight state, while the `webhosting.dispatch_management_jobs` tick recovers a task lost to a worker or broker restart. The node must report the matching resource identity and generation before the portal marks desired state active. Credential requests for a database that is still converging remain queued behind that structural job instead of being rejected by the customer interface. Customer file bytes use the separate bounded file-transfer protocol, never a management-job payload or a portal database row. Uploads declare their size and SHA-256 digest before streaming, downloads use a one-time node receipt, and the small text editor accepts only bounded UTF-8 tenant files. Signed download and edit URLs are excluded from activity, analytics, audit-path storage, and session recording and are served with no-store/no-referrer protections. New addon domains and subdomains use a readable `domains//public_html` document root. Immutable jobs created before that layout change retain their UUID-based document root during rolling deployment and replay, and an existing node binding is never renamed or copied implicitly. The live File Manager starts at the tenant's complete home, so customers navigate either layout without typing or controlling a host path. The text editor can edit a tenant-owned `.htaccess` inside a `public_html` tree. It never exposes a generated virtual host, `httpd.conf`, another tenant's tree, or any root-owned Apache file. The Files API remains a closed typed contract. Copy, archive creation/extraction, resumable upload, and permanent recursive purge are not exposed by this v1 contract. New database actions send only the customer's logical 1–32 character label. The node derives the physical database name from its root-owned account mapping as `_`; the portal no longer truncates labels. Immutable pre-upgrade actions that already contain a physical `name` remain valid for exact replay, and successful node results persist the observed physical name for later credential verification. Failed database and mail-domain creation rows expose an owned retry action. A retry locks the existing resource, proves its newest intent is the matching terminal failure, advances one generation, and queues a fresh typed action with the same immutable resource ID. An in-flight or newer intent suppresses retry. While management work is active, the control center shows a compact live list of real operation, target, and state labels. The bounded status endpoint polls at 750 ms, updates pending/submitted/running transitions in place, and uses a digest of active request identities to reload resource rows when membership changes—even when one completed job is replaced and the active count stays the same. The digest reveals no job or payload identifiers. Database and mailbox passwords use a separate credential-delivery lane, never `WebHostingManagementJob.request_payload`, results, task logs, or audit metadata. On **Generate credential** / **Rotate credential**, the portal locks the owned active resource, increments one credential generation, creates a request-specific RSA-3072 key pair, and Fernet-encrypts the private key. The authenticated HTTPS node API receives only the public key, immutable IDs, operation, generation, and a fourteen-minute mutation lease. This stays above the node's ten-minute pre-mutation safety floor while remaining below its fifteen-minute protocol maximum for clock-skew tolerance. The node returns the same sealed RSA-OAEP-SHA256 offer until the portal has transactionally stored Fernet ciphertext and explicitly acknowledges the exact request/envelope hashes. Lost consume or acknowledgement responses therefore replay safely; terminal node expiry fails and purges that generation so a new rotation can be requested instead of retrying forever. The customer then uses the POST-only **Reveal once** action. Ownership is checked again under a database lock; the recoverable encrypted credential is deleted and marked revealed before the standalone response is rendered. That response has no base layout, scripts, analytics, session recording, or third-party assets and sends no-store/CSP/no-referrer/nosniff headers. Its fixed `/client/web-hosting-credentials/` path is excluded from request activity, analytics, audit-path storage, and recordings. A second request cannot recover the password. The separate fifteen-minute reveal window starts only after the node acknowledges the durable portal ciphertext; an unrevealed credential is then purged when that window expires. Deployment requires migrations through `webhosting.0019`, a stable production `FIELD_ENCRYPTION_KEY`, the Celery worker, and Beat schedules `webhosting.dispatch_credential_deliveries` and `webhosting.expire_credential_deliveries`. A node must advertise both the qualified structural operations and its separately qualified matching credential rotation before MySQL, PostgreSQL, or mail controls become available. Do not mark PostgreSQL available until the node enforces and reads back storage quotas, and do not mark mail available until DNS/TLS/SMTP/IMAP, quota, relay, cross-tenant, and reputation staging passes. Mailbox credentials identify `mail.` for authenticated SMTP and IMAP. `email.` remains the separate HTTPS/webmail hostname; it is never substituted for the Postfix/Dovecot endpoint. ### Runtime Usage Billing maintenance aggregates elapsed runtime into usage records and account ledger debits. Hourly VPS runtime is charged from the plan usage meter. Floating IP reservations are charged separately while reserved, even when not assigned to a server. The maintenance command is dry-run by default: ```powershell .\.venv\Scripts\python manage.py run_billing_maintenance ``` Use `--live` to write usage, notices, suspensions, recoveries, and deletion queue changes: ```powershell .\.venv\Scripts\python manage.py run_billing_maintenance --live ``` Celery Beat runs `billing.run_billing_maintenance` every five minutes by default. Override `BILLING_MAINTENANCE_INTERVAL_SECONDS` when an environment needs a different cadence. Celery Beat also runs `billing.enforce_suspended_vm_power_state` every five minutes (`SUSPENDED_VM_CHECK_INTERVAL_SECONDS`). It reads live Proxmox state for suspended servers and queues a fresh stop plus disables autostart if a guest was turned on or has `onboot` enabled. Existing jobs and legitimate billing recovery are respected. The Scheduled Tasks console records each check and its repairs. Both billing maintenance and live provisioning must be enabled for repairs to reach Proxmox. To inspect the same checks manually, run `python manage.py reconcile_suspension_state --suspended-only`; add `--live` to queue repairs. The full fleet audit remains daily. The same maintenance run removes inactive orders that remain in `Awaiting payment` for three days, including their unpaid checkout invoices and payment attempts. Unpaid invoices that are not attached to those orders (checkout artifacts and leftover due-date rows) are removed after the same three days: credit billing never collects them, and the receivable is a negative ledger balance. Dry-run mode reports what would be removed. Override `BILLING_UNPAID_ORDER_RETENTION_DAYS` to change the retention period. Expected result: active billable resources produce ledger usage entries. The client live-usage graph and service breakdown explain why account credit changed for the selected month, with month history available for the latest 24 months. ### Credit Automation `BillingPolicy` controls the account-credit automation: * `low_credit_warning_days`: sends a daily low-credit warning when remaining credit would last this many days or fewer at the current burn (default 5). The copy and the escalation rungs (also in days) are editable under Settings > Marketing; when a low-credit sequence is enabled there, its highest rung widens this entry window. Accounts with a working saved card are not emailed until a charge fails; a successful auto-charge sends the payment receipt. See . * `auto_charge_threshold`: attempts a saved-method recharge when balance falls below the threshold. * `credit_suspend_balance_threshold`: suspends accounts at or below this balance. This value must be zero or negative. * `negative_credit_grace_days`: suspends accounts that remain negative for this many days. * `final_deletion_warning_days`: queues service deletion this many days after account suspension. Open invoices are ignored by credit automation. There is no invoice due-date dunning in the current credit-wallet flow. Expected result: accounts are suspended only because the wallet crossed the configured negative threshold or stayed negative through the configured grace period. ### Suspension, Recovery, And Deletion When credit automation suspends an account: 1. The billing account is marked suspended. 2. Active subscriptions are marked suspended. 3. Running, staged, provisioning, and stopped VMs are marked suspended locally. 4. Proxmox suspend jobs are queued for VMs that need provider-side pause. 5. Active shared-hosting services queue an exact, generation-bound node suspension job; their observed status changes only after node confirmation. 6. The suspension event records the reason, balance, policy, queued jobs, and deletion countdown. Stopped VMs do not need a provider-side pause job. Already-suspended accounts are repaired by maintenance if they are still negative and are missing the expected Proxmox suspend job. When a suspended account returns to a non-negative balance, maintenance unsuspends the account and subscriptions, queues Proxmox unsuspend jobs for VMs that were running, staged, or provisioning before billing suspension, and queues shared-hosting restoration. Hosting remains visibly suspended/pending until its node confirms the exact restoration generation. Maintenance also reconciles suspended accounts against the current credit policy on every run, so legacy invoice-past-due suspensions are cleared or replaced with current credit-policy suspension details. If the account remains suspended through the final deletion window, maintenance opens an operator termination review. VPS teardown still requires approval. A review containing live shared hosting cannot be approved yet: it remains queued with a `webhosting_deletion_unavailable` blocker and no service is deleted until immutable archive retention, clean-node restore, and two-stage node deletion have been implemented and qualified. Expected result: suspension is account-level, not just service-level. The user gets a warning and deletion countdown, all eligible VMs are paused in Proxmox, and recovery restores service when the account is funded. ### Floating IPs Floating IP reservation immediately creates an allocation and starts hourly usage. Assignment attaches the allocation to a server and syncs provider state. Release detaches provider state, stops customer billing for that allocation, and frees the address when cleanup is complete. Expected result: reserved IPs are billed while reserved, assignment/release operations are idempotent enough for retries, and release should succeed even when no server is assigned. ### Admin And Client Surfaces Client billing pages show wallet balance, credit top-up, saved payment methods, paid payment receipts, and selectable monthly usage with per-resource totals. Admin billing pages show paid payment receipts, billing accounts, payment attempts, provider configuration, usage, and automation history. Admin error handling should expose enough traceback/detail for staff to diagnose 500s without exposing secrets. Expected result: clients see what changed their credit balance. Staff can audit payment attempts, ledger entries, automation decisions, suspension reasons, and provider job state. ### Site Analytics Settings > Analytics holds one Google tag for the whole platform. Enter the GA4 measurement ID (`G-XXXXXXXXXX`) from Google Analytics > Admin > Data streams, switch it on, and gtag.js loads in the head of every full page render — the public marketing site, the client portal, the sign-in screens, and the operations console. Nothing is sent while the switch is off or the tag ID is blank. Uncheck **Track operations console pages** to keep internal staff activity out of Analytics; the public site, client portal, and customer ticket pages keep reporting. The bare 403/404/500 pages and email previews never carry the tag, and console navigation is partly client-side, so in-console clicks report fewer pageviews than full page loads. Expected result: one tag ID entered in the console tracks the entire site, and no deploy is needed to change or remove it. ### Operational Commands Run the provisioning worker loop locally with: ```powershell .\.venv\Scripts\python manage.py run_provisioning_jobs ``` Use `--live` only when the environment is configured to perform real provider actions: ```powershell .\.venv\Scripts\python manage.py run_provisioning_jobs --live ``` Run maintenance with `--live` from Celery Beat or an operator action in production. In local development, dry-run mode is useful for checking what would be charged, suspended, recovered, or queued for deletion. Expected result: web requests create orders and queue work; workers and maintenance commands perform provider actions and recurring billing changes. ## Railway Production For Railway, run one Docker service with one or more identical replicas: ``` LAYERONE_PROCESS_TYPE=all ``` That is also the Docker default when `LAYERONE_PROCESS_TYPE` is blank. Every replica starts: * Django web. * Celery worker. * Celery Beat in elected-leader mode. PostgreSQL advisory-lock leader election keeps exactly one Beat scheduler active across all `all` replicas. The other Beat processes remain hot standbys and take over automatically if the leader exits or loses its database session. The console's **Workers → Replica roles** table identifies that elected replica as **Primary** and lists every other live Beat replica as **Secondary**, using the Railway replica ID. Primary refers only to scheduled-task publishing: in `all` mode every primary and secondary still serves Django and runs a Celery worker. Billing maintenance has its own independent PostgreSQL singleton lock, so a duplicate Celery delivery or manual command cannot run a concurrent billing pass. You still need shared PostgreSQL and Redis attached to that same service. `REDIS_URL` only provides the broker; `LAYERONE_PROCESS_TYPE=all` is what starts the worker and beat processes that actually consume from it. Splitting the same image into separate services remains optional when you want to scale web and worker capacity independently: ``` web service: LAYERONE_PROCESS_TYPE=web worker service: LAYERONE_PROCESS_TYPE=worker beat service: LAYERONE_PROCESS_TYPE=beat # one active leader; extras standby ``` All Django processes need the same production `DATABASE_URL`, `REDIS_URL`, `SECRET_KEY`, `DJANGO_SETTINGS_MODULE=config.settings.production`, and app configuration. If the web service or a dedicated release job already runs `deploy_release`, set `RUN_DEPLOY_RELEASE=false` on split worker and beat services. In replicated `all` mode, concurrent startup release work is serialized by a separate PostgreSQL advisory lock. Leave `RUN_COLLECTSTATIC` unset unless you intentionally want to skip startup static verification. In `all` mode, Railway's HTTP healthcheck can stay on the replicated service: ``` /health/live/ ``` In split mode, apply that HTTP healthcheck only to the web service. Do not apply it to worker or beat services; Celery does not listen on `$PORT`, so an HTTP healthcheck will mark those services unhealthy even when the process is running correctly. See [Observability §11.6](https://docs.layeronecloud.com/platform/observability#id-116-multi-host-rules) for horizontal scaling and race-condition rules. ## Repository Access This is a private GitHub repository. Clone access requires an authorized LayerOne organization account and a configured GitHub credential or SSH key. Repository: ``` https://github.com/LayerOne-LLC/LayerOne.git ``` Clone with HTTPS: ```powershell git clone https://github.com/LayerOne-LLC/LayerOne.git cd LayerOne ``` Or clone with SSH: ```bash git clone git@github.com:LayerOne-LLC/LayerOne.git cd LayerOne ``` ## Production Configuration Production uses: * `DJANGO_SETTINGS_MODULE=config.settings.production` * `DATABASE_URL` for PostgreSQL. * `REDIS_URL` for cache and Celery. * `STATUS_MONITOR_WORKER_GROUP` only on mini agent hosts or trusted DB-backed monitor workers, not on the web server. * `TICKET_ATTACHMENT_MAX_BYTES` for database-backed ticket upload size limits. * `TICKETING_URL_PREFIX` (default `ticketing`) mounts the standalone staff ticketing app at `//`. One path segment; a value that collides with an existing mount fails the system check at startup. * `APP_BASE_URL=https://layeronecloud.com` on every production web, worker, and Beat service. Production defaults to this origin when unset or blank and replaces retired `pulsar67.com` / `www.pulsar67.com` origins with it. Explicit origins for other deployments remain configurable; nonlocal email links use the canonical LayerOne origin. Local development defaults to `http://127.0.0.1:8000`. * `AGENT_API_PSK` only as a temporary compatibility fallback for pre-migration agents; new agents use keys created under Settings. * `AGENT_ONLINE_SECONDS=300` so agents are treated as down if they have not checked in within five minutes. * Email delivery: `EMAIL_CLOUDFLARE_ACCOUNT_ID`, `EMAIL_CLOUDFLARE_API_TOKEN` (token needs the *Email Sending: Edit* permission), and `DEFAULT_FROM_EMAIL`. `deploy_release` syncs them onto the email delivery configuration and enables it; the sender domain must first be onboarded under Cloudflare's Compute > Email Service > Email Sending. Cloudflare's REST API is the only transport — there is no SMTP path, and with no enabled configuration production records outbound mail as failed instead of sending it. * `EMAIL_TASKS_ALWAYS_EAGER=true` for single-service web deployments so login codes and ticket notifications send immediately. Set it to `false` only when a Celery worker is running. * `SERVICE_CHECKIN_REGION` for the default service check-in region label. * `SERVICE_KILL_ON_SIGNAL=true` so checked-in services exit after an admin kill request. * `BILLING_PROVISIONING_LIVE=true` in production to perform real Proxmox actions (local dev defaults to dry-run). * Optional ISO library bucket on the Proxmox connection (Infrastructure / Connections): S3 API endpoint, bucket, region, object prefix, and write-only access keys. Keys are Fernet-encrypted on the connection row, not environment variables. That is the same private bucket rclone mounts as `l1-isos`. When set, the browser PUTs installer ISOs to `template/iso/` and Proxmox lists the file on the mount. Leave the bucket fields blank to keep the slower Postgres chunk path. Bucket CORS must allow PUT from `https://layeronecloud.com` and expose `ETag`. * `PROVISIONING_STALE_LOCK_TIMEOUT_SECONDS=900` controls when a RUNNING provisioning job whose worker disappeared is returned to the queue. Recovery preserves successful steps and resumes unfinished work. * `PROVISIONING_AUTO_RETRY_MAX_RETRIES=3` and `PROVISIONING_AUTO_RETRY_DELAY_SECONDS=300` keep an initial VPS or web-hosting deployment in the queue for up to three automatic retries after a terminal attempt, waiting five minutes between attempts. After that it becomes failed operator work; manual retries remain available. * `PROXMOX_STORAGE_LOCK_MAX_ATTEMPTS=500` and `PROXMOX_STORAGE_LOCK_RETRY_DELAY_SECONDS=30` automatically return transient Proxmox clone storage-lock timeouts to the end of the provisioning queue for up to 500 total attempts, waiting at least 30 seconds between attempts. * `PROXMOX_VM_LOCK_MAX_ATTEMPTS=8` and `PROXMOX_VM_LOCK_RETRY_DELAY_SECONDS=5` return `VM is locked (create)` (and other guest locks) to the queue after five seconds, for up to eight job attempts. The worker also waits up to `PROXMOX_VM_LOCK_IN_STEP_WAIT_SECONDS=45` (polling every `PROXMOX_VM_LOCK_POLL_SECONDS=2`) before failing the step, so a lock that clears in a few seconds never leaves the job. * Healthy web-hosting account actions become due for observation after `WEBHOSTING_PROVISIONING_POLL_INTERVAL_SECONDS=5` and are picked up by the shared provisioning dispatcher on its next tick. Pending/running responses preserve the same action and completed steps without consuming a failure retry. `WEBHOSTING_PROVISIONING_POLL_TIMEOUT_SECONDS=900` bounds observation from that action's original submission (or creation before submission); reaching the deadline or encountering node errors uses the existing bounded `PROVISIONING_AUTO_RETRY_MAX_RETRIES` and `PROVISIONING_AUTO_RETRY_DELAY_SECONDS` policy. * `WEBHOSTING_MANAGEMENT_DISPATCH_INTERVAL_SECONDS=15` controls recovery of durable customer-management intents whose immediate broker publish failed. Healthy management actions are observed every two seconds with one leased step per job; duplicate deliveries cannot create extra polling chains. Transport failures back off, and resource successors wake as soon as their predecessor finishes. * `WEBHOSTING_ACCOUNT_ACCESS_CACHE_SECONDS=30` reuses a verified account read across hosting workspaces while local ownership, node binding, and desired state remain unchanged. Set it to `0` to check on every page load. * `WEBHOSTING_MANAGEMENT_SUBMITTED_NO_PROGRESS_TIMEOUT_SECONDS=120` fails a submitted node action when its state, event stream, and node-reported update timestamp have not advanced for two minutes. Identical HTTP poll responses do not reset this deadline. * `WEBHOSTING_MANAGEMENT_RUNNING_NO_PROGRESS_TIMEOUT_SECONDS=120` applies the same two-minute no-progress rule to running actions. It remains separately configurable for node releases that emit bounded heartbeats during longer provider or backup work. * `WEBHOSTING_NODE_SYNC_INTERVAL_SECONDS=30` continuously refreshes enabled CloudLinux node health and API inventory and wakes reservations waiting for placement. `WEBHOSTING_NODE_SYNC_LIMIT=10` bounds nodes per tick. * `WEBHOSTING_CERTIFICATE_RENEWAL_INTERVAL_SECONDS=3600` controls the scan that gives customer-domain certificates a fresh immutable renewal generation before expiry. * `WEBHOSTING_CUSTOMER_DNS_DISPATCH_INTERVAL_SECONDS=15` controls recovery of durable portal-to-Cloudflare customer DNS jobs. The per-zone customer DNS authorization remains off until an operator enables it in the console. ### Application platform (GitHub deploys) The application platform runs customer apps as microvms. Two pieces of setup are separate on purpose, because they fail differently: * **Hardware.** Mark at least one Proxmox connection as **MicroVM** under Infrastructure / Connections. A connection runs KVM servers or microvms, never both, and KVM is the default, so nothing is marked until you do it. Until one is, applications can be created but have nowhere to run and the interface says so. * **GitHub.** Register an OAuth app whose callback is `https:///client/apps/github/callback/`, then set: * `PAAS_GITHUB_CLIENT_ID` and `PAAS_GITHUB_CLIENT_SECRET`. Absent, the Connect GitHub surfaces do not appear at all rather than appearing and failing. * `PAAS_GITHUB_WEBHOOK_SECRET` enables build-on-push. Absent, customers can still connect and pick repositories; they just cannot build automatically, and **unsigned deliveries to `/paas/github/push/` are refused rather than trusted**. * `PAAS_GITHUB_TIMEOUT_SECONDS=10` bounds outbound calls to api.github.com. The OAuth app asks for the `repo` scope, which covers every repository the authorizing user can reach. If that is broader than you want to hold, move to a GitHub App before launch — see . ### Web-hosting control-plane deployment Use this order when bringing the portal and a CloudLinux node online: 1. Set a stable production `FIELD_ENCRYPTION_KEY`, `DATABASE_URL`, `REDIS_URL`, `SECRET_KEY`, and `BILLING_PROVISIONING_LIVE=true` on every web, worker, and Beat process. Do not rotate the encryption key after node, DNS, or credential secrets have been stored without an explicit data-rewrap procedure. 2. Deploy the release and migrations, then restart the web process, every Celery worker, and the single Beat process: ``` python manage.py deploy_release python manage.py deployment_check --strict ``` 3. Install or update the separate CloudLinux node using its own README. Copy the one-time key generated by the node into **Web hosting → Nodes** together with the exact configured node ID and HTTPS API URL. The scheduled node-sync task tests enabled nodes automatically; **Test connection** remains an immediate operator diagnostic. This integration uses the bearer API key and normal server TLS validation; it does not use an mTLS client certificate or an application CIDR setting. 4. Confirm the node is enabled and **Online**. Existing LET Beta reservations in `Awaiting node` are attached and queued by the first successful automatic node sync; the shared provisioning dispatcher is their recovery path. 5. Confirm the node reports the expected fixed typed API inventory. Inventory flags are informational and do not gate placement or customer controls. 6. Submit one disposable hosting account and verify its provisioning result, LVE/CageFS limits, PHP Selector, Apache isolation, and customer control-center ownership before opening the plan to customers. Keep worker and Beat running after deployment. Web-hosting account placement, management reconciliation, certificate renewal, retries, and hourly metering are durable asynchronous workflows; restarting only the web process leaves them queued until a worker and the scheduled recovery tasks are available. Status monitor worker groups: * `group-1`: Florida, USA * `group-2`: Frankfurt, Germany * `group-3`: Virginia, USA Service check-in region labels: * `Florida, USA` * `Frankfurt, Germany` * `Virginia, USA` The web server is the control plane: public website, console, agent API, and status data. It should not run monitoring checks or Looking Glass commands. Leave `STATUS_MONITOR_WORKER_GROUP` blank on web, beat, and ordinary worker containers. Mini agents are the normal monitoring and Looking Glass path. Create and name each one under **Settings → Monitoring agents**; every agent receives its own one-time API key. The authenticated record controls its worker group and region. Agents claim monitoring and Looking Glass work and submit results through the REST API over outbound connections; they do not need a public Looking Glass URL or inbound listener. DB-backed monitor workers still exist for trusted internal environments. They authenticate through the shared PostgreSQL database credentials in `DATABASE_URL`, claim checks with database row locks, and write results directly to the database. The packaged DB-backed monitor worker command is: ``` python manage.py run_due_monitors --loop ``` Set `DATABASE_URL`, `DJANGO_SETTINGS_MODULE`, `SECRET_KEY`, and `STATUS_MONITOR_WORKER_GROUP` on that container or host. Production settings also expect `REDIS_URL`. For Ubuntu hosts that should run only monitoring and Looking Glass without database access, install the separately distributed standalone agent release with its per-agent key. It defaults the API to `https://layeronecloud.com`. Mini agents do not need `DATABASE_URL`, `SECRET_KEY`, `DJANGO_SETTINGS_MODULE`, Redis, email credentials, worker-group settings, or production Django settings. See . Django-managed web, Celery, service-checkin, and DB-backed monitor worker processes write online service check-ins directly through the shared database. Mini agents check in through the REST API. For Django-managed processes and mini agents, `SERVICE_KILL_ON_SIGNAL=true` lets the process exit after acknowledging a pending kill signal. Equivalent commands if you use explicit Railway start commands instead of `LAYERONE_PROCESS_TYPE`: ``` web: uvicorn config.asgi:application --host 0.0.0.0 --port $PORT --proxy-headers --forwarded-allow-ips='*' worker: celery -A config worker -l info beat: celery -A config beat -l info ``` When deploying web-hosting changes, redeploy or restart all three processes—not only `web`. The worker and Beat use the shared `billing.run_pending_provisioning_jobs` dispatcher for VM and initial hosting- account placement. Active-service customer controls use `webhosting.reconcile_management_job`, which schedules its own next poll until the node result is terminal; the 15-second `webhosting.dispatch_management_jobs` schedule recovers missed or interrupted tasks. `webhosting.renew_domain_certificates` queues qualified AutoSSL renewals independently of domain routing state. Keep the five-minute `webhosting.meter_hourly_usage` schedule for completed post-trial hours. After the node is enabled and Online, the `webhosting.sync_nodes` task transactionally attaches it to old `Awaiting node` reservations, backfills their generated identity and shared `ProvisioningJob`, and marks them Queued. **Web hosting → Test connection** invokes the same path on demand, but it is not required for provisioning. The shared dispatcher's scheduled pass is the recovery path if an immediate worker handoff is briefly unavailable. Capability rows describe the installed release's typed API groups; they are not placement or control-admission authority. ### Hosted-mail DNS setup Hosted-mail DNS uses its own Cloudflare token. Do not reuse the node API key, the website email-delivery token, or a global API key. 1. Deploy migrations `webhosting.0013` and `webhosting.0014` with the same stable production `FIELD_ENCRYPTION_KEY` used by every web/worker process. 2. In Cloudflare, create an API token with only **Zone Read** and **DNS Write** for the exact authoritative zones that LayerOne will manage. 3. Open **Web hosting → Hosted-mail DNS** in the operations console. Paste the token once, enable the provider, and add each approved zone name together with its exact 32-character Cloudflare zone ID. 4. Let the node reconcile `ensure_mail_domain`. The node, not the browser, generates the DKIM key and returns a closed public DNS intent for `mail.`, `email.`, MX, SPF, DKIM, DMARC, and optional client discovery records. 5. Use **Publish approved records** for that mail domain. The portal applies only that validated intent, marks every record with the domain's immutable LayerOne ownership ID, rejects unmanaged conflicts, removes only stale LayerOne-owned records, and performs an exact Cloudflare readback. 6. Run the node's hosted-mail qualification again after public DNS propagates. DNS publication alone never marks mail ready: current TLS/SNI, Postfix/Dovecot authentication, mailbox quota, no-open-relay, Rspamd, outbound-policy, cross-tenant, and maintained Roundcube proofs must all pass. 7. After that root-only node verification succeeds, use **Recheck production readiness** for the pending mail domain. This advances its immutable generation and submits a fresh `ensure_mail_domain` job. Only the node's current, owner- and DNS-intent-bound `active` observation activates the mail domain and exposes mailbox or alias creation. Replaying the original `pending_dns` job cannot activate it. Changing the node-generated DNS intent clears the previous provider receipt and every old production proof; an exact identical retry is idempotent. The customer never supplies arbitrary DNS record types, values, provider zone IDs, or provider credentials. ### Web-hosting backup compatibility Customer backup tools and mutation routes are intentionally unavailable in the current portal. Existing backup resources, plan fields, historical management jobs, admin inspection, and rolling-deployment reconciliation remain intact so removing the unfinished customer surface does not erase operational history or break already-submitted work. Migration `webhosting.0018` retains explicit backup metadata for that compatibility boundary: * LET Beta: one retained backup, seven-day maximum retention, one creation per 24 hours. * Starter: two retained backups, seven-day maximum retention, one creation per 24 hours. * Developer: five retained backups, 14-day maximum retention, one creation per six hours. * Business: ten retained backups, 30-day maximum retention, one creation per hour. No customer backup creation, restore, retention deletion, or in-place account replacement is exposed. Reintroducing that surface requires a separately authorized design and proven restore/rollback workflow. ### Customer DNS-zone setup Customer DNS is a portal-to-Cloudflare workflow and is independent of node management capabilities. It reuses the write-only encrypted provider token and approved-zone mapping above, but every approved zone has a separate **Allow customer DNS management** switch. Migration `webhosting.0017` creates the desired-state/job journal and leaves that switch off for existing and new zones. 1. Deploy migrations, then restart the Django web process, every Celery worker, and the single Beat process. Keep the same stable `FIELD_ENCRYPTION_KEY` on each process. 2. Grant the scoped Cloudflare token only Zone Read, DNS Read and DNS Write for the exact approved zones. DNSSEC Read is optional for the operator-only observation button. Never reuse the node API key, global Cloudflare API key, or website email-delivery token. 3. In **Web hosting → Hosted-mail DNS**, verify the exact canonical parent zone and 32-character Cloudflare zone ID. Leave customer management disabled while staging qualification is incomplete. 4. On a disposable staging domain, exercise A, AAAA, CNAME, TXT, MX and CAA create/update/delete, exact readback, retry after a deliberately interrupted response, stale-generation refusal, delegation/CNAME conflict refusal, hosted-mail/ACME protection, and cross-account ownership denial. 5. After those live provider checks pass, enable **Allow customer DNS management** only on the qualified zone. Provider enablement or hosted-mail publication alone does not expose customer mutation controls. The customer UI never receives the provider token, Cloudflare zone/record ID, raw provider response, proxy controls, NS/SOA/delegation operations or an arbitrary provider payload. Celery persists only canonical public intent and local bounded error codes; missed immediate dispatches are recovered by `webhosting.dispatch_customer_dns_jobs`. The console rejects provider or zone edits while any customer DNS job is pending, running or retrying. It also keeps the canonical zone name and Cloudflare zone ID fixed while a non-deleted customer record remains, so an already-applied provider record cannot be orphaned by a control-plane remap. Run before or during deployment: ``` python manage.py collectstatic --noinput python manage.py deploy_release python manage.py deployment_check --strict ``` Railway uses the repository `Dockerfile`, which runs Django `collectstatic`. The Docker build compiles the committed Tailwind source with the checksummed standalone CLI and rebuilds the marketing CSS bundle before `collectstatic`. `deploy_release` runs database migrations, initial site/catalog seeding, and environment-backed email configuration sync. Public marketing pages live in Django templates and CMS records. Use `/health/` or `/health/live/` for process liveness and `/health/ready/` for dependency readiness. ## Current Scope Included now: * Django project with modular first-party apps for each platform subsystem. * Local (`config.settings.local`) and production settings. * SQLite local fallback; PostgreSQL + Redis in production. * Railway-ready Dockerfile and process commands. * Bootstrap 5.3-backed design system for console, client portal, and marketing site. * Email + password authentication with optional email-code verification flow. * Custom user model with Admin and Client access levels. * Stay-logged-in sessions and session revocation records. * Public marketing site with CMS pages and blog. * Client billing portal at `/client/` (orders, services, credit, VNC console). * Full VPS billing: catalog, orders, credit ledger, Stripe webhooks, Proxmox provisioning, IPAM, VM lifecycle, floating IPs, billing maintenance. * Shared web-hosting pricing and LET Beta reservations with a 30-day trial (no credit card or credits required to deploy), $1.17/month hourly-billed continuation from account credit, or automatic cancellation if the account has no credit when the trial ends, and plan-specific CloudLinux limits (128 MB memory for LET Beta). * Durable CloudLinux account placement plus customer control-center resources for domains/subdomains, reviewed Apache and `.htaccess` policies, PHP Selector, Python/Node applications, MySQL/PostgreSQL, mail domains/mailboxes/ aliases, and tenant-file metadata and bounded content-transfer operations. * Typed Python/WSGI, Django, and Node release management with bounded USTAR artifact upload, immutable candidate/active/rollback generations, separately sealed environment-secret delivery, and one-time sanitized log tails. Django schema migration remains closed until a backup-bound step-up flow exists. The portal commits an exact artifact ID, expiry, size, digest, dependency digest, release, and generation before node I/O. A lost create/content response is recovered by exact replay or GET; changed bytes cannot adopt the intent, and a terminal or pruned artifact requires a fresh release generation. * Structured hourly/daily/weekly/monthly PHP, Python, and Node scheduled tasks bound to an owned domain or active application; customers never submit cron text, executable paths, shell syntax, environment values, or output content. * Persisted backup resources, plan metadata, and historical job reconciliation retained for admin and rolling-deployment compatibility; no customer backup tool or mutation route is exposed. * Write-only, separately encrypted Cloudflare hosted-mail DNS configuration with approved-zone mappings, exact provider-independent DNS intent, LayerOne-owned record reconciliation, conflict detection, and provider readback. DNS publication does not bypass the independent mail production gate. * Default-off customer DNS management for active owned hosting domains, with typed A/AAAA/CNAME/TXT/MX/CAA intent, protected mail/ACME/system records, encrypted provider receipts, durable generation-bound jobs, exact readback, and operator-observed DNSSEC status. * One-time sealed database and mailbox credential delivery with customer password step-up for reveal actions. * Separate customer-domain AutoSSL state and renewal jobs, so HTTP routing, certificate expiry, and HTTPS readiness cannot be conflated in the portal. * Persisted LayerOne site and public status records. * Site-scoped dashboard routing. * Client and admin ticketing with encrypted ticket secrets and attachments. * Admin and Client areas derived from the route, with an optional dedicated administrator hostname configured in Settings. * Append-only audit event foundation. * Manual incident lifecycle with controlled public status publication. * Infrastructure inventory with topology, interfaces, tags, and retirement workflows. * Monitoring foundation with HTTP/TCP/TLS/ping checks and uptime summaries. * Mini agent and DB-backed monitor worker execution paths. * JSON health, liveness, and readiness checks. * Deployment diagnostic management command and Docker healthcheck. * Admin service check-ins with kill-signal support. * Client email notifications for admin ticket replies. * Full test suite, all-pages smoke script, and navigation link audit script. Disposable CloudLinux image tests remain an engineering release check, not runtime feature authority. Online nodes accept every implemented typed API operation; each mutation independently validates ownership and reads back its effective state. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.layeronecloud.com/platform/setup.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.