> For the complete documentation index, see [llms.txt](https://docs.layeronecloud.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.layeronecloud.com/platform/observability.md).

# Observability and operations

audit.AuditEvent — actor, action, category, target, site, request metadata, before, after and metadata payloads.

## 11.1 Audit log <a href="#id-111-audit-log" id="id-111-audit-log"></a>

`audit.AuditEvent` — actor, action, category, target, site, request metadata, `before`, `after` and `metadata` payloads. **Append-only application behaviour**, read-only in Django admin. `record_audit_event()` writes **after** the database transaction commits (`transaction.on_commit`), so a rolled-back change does not create a misleading committed audit row. Writes are plain inserts with no application-local counters.

Every event carries `actor_mode`, so an admin write can no longer be logged with `actor_mode=client` ([1.6](/platform/identity/access-modes.md#id-16-access-modes-and-the-administrator-site)).

**Data safety.** Audit events avoid storing full bodies: ticket and incident events store structured operational metadata (ticket number, status, priority, type, site slug, message **length**, internal/public flag) rather than reply text; inventory events record note **length**; monitoring events record target **length** rather than raw targets; the DDoS/DNS/node families record identifiers, generation, action, outcome and local error codes only. Credential-bearing paths are redacted through `apps/core/credential_paths.py`.

The console log at `/console/audit/` filters by actor, category, action and success/failure, and paginates. Audit rows route through `audit_target_url` (`apps/audit/templatetags/audit_links.py`) rather than an `{% if %}` chain — eleven target types across six apps would otherwise drift page by page — and it **refuses to link a `.deleted` action**, since the row is a tombstone and the link is a guaranteed 404. Client and client-mode admin users cannot access the audit log; it is default-deny for Support.

Adding a value to a `TextChoices` used by a model field (including `AuditAction`) **requires a migration** — run `manage.py makemigrations`, do not hand-write it.

## 11.2 Session recordings <a href="#id-112-session-recordings" id="id-112-session-recordings"></a>

First-party session replay of the public website and client portal using open-source rrweb (`@rrweb/record` + `rrweb-player`, MIT, vendored). **Data never leaves LayerOne.** The player lives under `/console/recordings/` as a Logs tab, **off `SUPPORT_CONSOLE_ROUTES`**, and capture is **disabled by default** until Settings → Recordings is flipped.

* **Recorder scope:** the public site, `/client/` (except credential prefixes), and guest `/account/sign-in/` / `/account/register/` only. Injected from `templates/includes/session_recorder.html` via `_head_scripts.html`.
* **Ingest:** `POST /internal/recordings/ingest/` with **CSRF + a signed `l1_sr` cookie**. Not the client API, and **not `@csrf_exempt`.**
* **Never recorded:** `/console/` (the entire operator shell), credential-bearing paths, `/account/two-factor/` and the rest of `/account/` other than sign-in/register, `/internal/`, `/api/`, `/health`, `/static/`, `/media/`, Stripe card fields (iframe), typed passwords/MFA codes/API keys (masked), and the VNC framebuffer (`recordCanvas: false`, and console routes are denied anyway).
* **Masking:** `maskAllInputs` hides form **values**. **Text nodes that display secrets need `rr-mask` as well** — root password, API key one-shot, recovery codes, ticket secrets. Didit hosted-session links carry `rr-block` because their `href` contains a bearer token and **rrweb text masking does not sanitize attributes.** Tests pin the class on those nodes and pin the JS deny-prefix list to the Python tuples.
* **Sampling and caps:** checkout prefixes always record when `always_record_checkout` (default on); other `/client/` always records when `always_record_client` (default on); marketing pages use `sha256(ip:utc-date) % 100 < sample_rate_percent` (default 20), sticky per IP per UTC day, and **a valid `l1_sr` cookie continues a session through guest sign-in even when marketing sampling would have said no.** Bot UA denylist, Cloudflare bot score / verified-bot headers when present, 30 POSTs/IP/minute, max 3 active recordings per IP, 900 s duration, 2 MiB compressed, 1 MiB request bodies. Client batches are serialized and retried with idempotency keys, and an idle visible tab sends a **15-second heartbeat** so the live view reflects presence even when the DOM is not changing. Active rows with no heartbeat for two minutes are finalized as Completed by a minute-level Beat task; a late heartbeat or batch can **reactivate** the same recording until its duration or size cap is reached, so background-tab throttling does not split the session.
* **Replay delivery:** the console events endpoint is `no-store` and returns at most 25 compressed chunks per response, with `has_more` plus `after_seq` letting the player drain a long recording without one oversized JSON response before switching into rrweb's live mode. A short reconnect window tolerates a missed heartbeat before finalizing as a normal replay. Audit `recording.viewed` fires on the **player page GET**, not on events polls.
* **Tenancy:** authenticated recordings retain the selected `billing_account` alongside the real signed-in user, and **a tenant change ends the current replay and requests a fresh snapshot**, just as changing users does. Admin account links use the recorded tenant, so reviewing an organization member's replay does not link to their personal billing account.
* **Retention:** `recordings.purge_old_recordings` daily, default 14 days, CASCADE deletes chunks. PostgreSQL stores gzip JSON arrays — no filesystem, no S3 in v1.
* Console review adds period/state/duration filters, observed path and capture summaries, and bounded related captures. **Paths describe unique first-seen observations, not a timed journey or proof of conversion.**

Non-goals in v1: heatmaps, rage-click, recording `/console/`, R2/S3, a consent banner, third-party SaaS, `@rrweb/packer`, shareable replay URLs.

## 11.3 Health, deployment checks and error reporting <a href="#id-113-health-deployment-checks-and-error-reporting" id="id-113-health-deployment-checks-and-error-reporting"></a>

* `/health/live/` — process liveness, no dependency checks. **This is the Railway healthcheck** and the Docker `HEALTHCHECK`.
* `/health/ready/` — database and cache readiness.
* `/health/` — liveness-compatible alias for platform startup checks.
* `manage.py deployment_check [--strict]` — database, cache, migration, static CSS, default-site and runtime-setting diagnostics; `--strict` exits non-zero on launch-blocking failures.
* `manage.py billing_launch_check [--strict]` — enabled payment provider configs, enabled Proxmox clusters, active templates, failed VMs, failed payment attempts and open invoices.

**Error reporting.** `ServerErrorReport` captures 5xx detail for staff, and `apps/core/error_reporting.py` **must not capture credential-bearing paths** — `/console/billing/console/` is registered there as well as in `apps/core/credential_paths.py`. Admin error handling should expose enough traceback for staff to diagnose a 500 **without exposing secrets**. The bare `403.html`, `404.html` and `500.html` templates are self-contained by design (inline CSS, no static files, no DB reads) so they render when the platform is broken.

## 11.4 `deploy_release` <a href="#id-114-deploy_release" id="id-114-deploy_release"></a>

`manage.py deploy_release` runs under a **PostgreSQL advisory lock** so concurrent replicas serialize, and performs: `makemigrations` → `migrate` → `seed_initial_site` → catalog seeding → the bundled public website import → `ensure_all_automations` → email configuration sync from the environment → the knowledge index rebuild.

**The website import treats the bundled development content as the source of truth and overwrites matching production website pages and blog posts on each deploy** ([2.2](/platform/public-site.md#id-22-cms-pages-and-blog)).

By default every Docker container startup also runs `collectstatic --noinput` before the configured command. Set `RUN_DEPLOY_RELEASE=false` / `RUN_COLLECTSTATIC=false` only when an external release job owns those steps. The image also runs `collectstatic` during **build** as an early sanity check for required static assets, and the Docker build compiles the committed Tailwind source with the checksummed standalone CLI and rebuilds the marketing CSS bundle before `collectstatic`.

## 11.5 Processes, workers and scheduled tasks <a href="#id-115-processes-workers-and-scheduled-tasks" id="id-115-processes-workers-and-scheduled-tasks"></a>

`LAYERONE_PROCESS_TYPE` selects `all` (default), `web`, `worker`, `beat` or `remote-monitor`. Equivalent explicit commands:

```
web:    uvicorn config.asgi:application --host 0.0.0.0 --port $PORT --proxy-headers --forwarded-allow-ips='*'
worker: celery -A config worker -l info
beat:   celery -A config beat -l info
```

**Celery Beat uses a PostgreSQL advisory-lock election.** Standby schedulers do not publish, and a promoted scheduler **realigns from shared task history before its first tick**. The console's **Workers → Replica roles** table labels the elected replica **Primary** and every other live Beat replica **Secondary** using the Railway replica ID — *Primary* refers only to scheduled-task publishing; in `all` mode every replica still serves Django and runs a worker.

**Billing maintenance has its own independent singleton lock**, so a duplicate Celery delivery or a manual command cannot run a concurrent billing pass.

**Managed process check-ins are role-qualified** (`:web`, `:worker`, `:beat`), which is what stopped a combined container's web heartbeat overwriting a live worker's record and reporting zero workers. Worker prefork children share the worker identity; local processes without a Railway identity keep their hostname/PID identity.

When deploying web-hosting changes, **redeploy or restart all three processes** — placement, management reconciliation, certificate renewal, retries and hourly metering are durable asynchronous workflows, so restarting only `web` leaves them queued. See [Appendix C](/platform/reference/scheduled-tasks.md) for the schedule inventory.

**Service check-ins and kill signals.** `ServiceInstance` is the registry of online processes, with safe environment snapshots built from an **allowlist plus private-key pattern blocking** — never a raw environment dump. Web processes check in through throttled middleware; Celery and management commands have their own paths; mini agents check in over the REST API. The admin dashboard shows online status and safe env values and can request a kill; `SERVICE_KILL_ON_SIGNAL=true` (default on for Django-managed processes) makes a process exit after acknowledging a pending kill. `AGENT_ONLINE_SECONDS=300` decides when an agent reads as down; `SERVICE_CHECKIN_REGION` labels the default region.

## 11.6 Multi-host rules <a href="#id-116-multi-host-rules" id="id-116-multi-host-rules"></a>

Web containers are stateless and scale horizontally; shared state lives in managed services. **Production must use shared PostgreSQL (relational data, sessions, tickets, audit) and shared Redis (cache, throttle counters, Celery broker and result backend). Do not use SQLite, LocMem cache or Celery eager mode in production** — those are local development fallbacks.

Safe to run with multiple replicas: combined `all` containers, web containers, worker containers, remote monitor workers. **Run exactly one active** Celery Beat scheduler (enforced by the election) and one external deployment release job if `RUN_DEPLOY_RELEASE=false`.

Release order: build image → start or roll web → start workers → start exactly one Beat → start remote monitor workers per region.

Race-condition protections already in place: partial unique constraints and transactional replacement on login challenges; `cache.add`/`cache.incr` for cooldowns and counters (atomic on Redis); database-stored user sessions checked by middleware; one default status page per site and unique status components by (page, slug) enforced in the database; seeding in a transaction under an advisory lock and idempotent through unique slugs; monitor claims by conditional update with short leases; remote monitor workers authenticated by database credentials **and** still requiring an active per-monitor claim token; monitor state and monitor-created incidents in one transaction; the Beat election; and the billing maintenance lock.

Operational rules: one `SECRET_KEY`, one `DATABASE_URL` and one `REDIS_URL` shared per environment; `DJANGO_SETTINGS_MODULE=config.settings.production`; `STATUS_MONITOR_WORKER_GROUP` **only** on monitor worker hosts; healthchecks against `/health/` or `/health/live/` (readiness on `/health/ready/`); `EMAIL_TASKS_ALWAYS_EAGER=true` for single-service web deployments and `false` only when a worker is processing the same queue; ticket attachments are database-backed; in-flight ISO pieces are database-backed unless the Proxmox connection has ISO library bucket credentials, in which case the rclone-backed library bucket is the replica-safe object and Postgres chunks are unused for that import; logs to stdout/stderr; **do not use sticky sessions as a correctness requirement** — any replica should be able to serve the next request.

Future hardening already identified: idempotency keys for worker result submission, and webhook delivery idempotency by provider event ID beyond the current per-provider event table.

***


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.layeronecloud.com/platform/observability.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.