> For the complete documentation index, see [llms.txt](https://docs.layeronecloud.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.layeronecloud.com/network-monitor/operations/retention-and-memory.md). # Retention and data-store memory Two windows, both enforced by the hourly maintain\_metrics job and both defined in server/monitoring/retention.py: ## Retention Two windows, both enforced by the hourly `maintain_metrics` job and both defined in `server/monitoring/retention.py`: | Kept for | What | | ------------ | -------------------------------------------------------------------------------------------------------------------------------- | | Three months | minute traffic, application facts and coverage, WAN probe and availability history, IP bindings, events, resolved alert episodes | | One year | DDoS incidents, their site-level correlations, suspects and destination rows, and their `ddos_detected` events | Past its window, data is deleted from the database: whole daily partitions are dropped where the table is partitioned, rows are deleted where it is not, and physical removal can lag by up to one maintenance interval. Keep the job scheduled hourly on custom deployments, or nothing is ever removed. These windows are deliberately much shorter and are not covered by the above: * Exact client/service endpoint pairs keep 24 hours. They are per connection per minute and at three months would dominate storage; the ranking and destination views they feed are built from the three-month facts instead. * Open-port observations and scan-delivery receipts keep seven days. Partial attempts do not extend the original timestamp of an older observation. * Packet-capture evidence files keep `PCAP_RETENTION_DAYS` (seven days). The incident that names them is kept for the full year, so an attack record outlives its packets; the console marks a capture unavailable once expired. When an incident finally passes a year, any remaining file is deleted from object storage before the row that names it, and a bucket that cannot be reached leaves both in place to be retried rather than orphaning the file. Extending ordinary history from one month to three roughly triples the fact tables, which dominate the database. Check `pg_database_size` against the volume before assuming there is room, and see *When the server refuses uploads* for what a full volume looks like. Sensors may backfill up to the ordinary window: telemetry older than three months is refused with `minute telemetry is older than the retention window` and the sensor replaces it with an exact gap, because there is no partition left to hold it. ## Data-store memory Redis keeps two disposable workloads: short-lived Django Channels keys and the `l1:live:` chart-repair streams. Durable telemetry never touches it; uploads wait in the PostgreSQL inbox table (`monitoring_telemetryinbox`). Retained live frames contain only the total and traffic dimensions used by the one-hour chart and are Zstandard-compressed. Each active site stream is trimmed by age and by an approximate count cap, and its TTL removes the key after the site stops publishing. The bounds are configurable, but are also user-visible behavior: | Django setting | Environment variable | Default | Effect | | -------------------------------- | ------------------------- | ------: | ---------------------------------------------------- | | `LIVE_HISTORY_RETENTION_SECONDS` | `L1_LIVE_HISTORY_SECONDS` | `3600` | Age limit and inactive-key TTL for live chart repair | | `LIVE_HISTORY_MAXLEN` | `L1_LIVE_HISTORY_MAXLEN` | `20000` | Per-site safety cap and maximum history read | Keep both values identical on `Web` and any separate telemetry worker service. Before changing them, measure each site's frame rate and `XLEN`: lowering the retention shortens chart recovery after a reload, while setting the count below the frames produced during that window truncates it early. The dashboard's 24-hour application ranking is read from PostgreSQL and refreshes every five minutes; it does not depend on, or extend, this Redis retention. The rolling default window is held in the web process for the seconds set under **Agents & Settings > Telemetry processing** (90 by default) so several open tabs do not each re-rank a day of facts; the authority decision behind it is made once per request from the coverage markers rather than per usage row, which keeps a busy site's day of facts answerable in seconds instead of timing out the dashboard. Start a Redis investigation by separating allocator footprint from live data and identifying the key family using the memory: ```sh redis-cli -u "$REDIS_URL" INFO memory redis-cli -u "$REDIS_URL" INFO keyspace redis-cli -u "$REDIS_URL" --bigkeys -i 0.05 ``` `--bigkeys` is SCAN-based but should still be run during a quiet period on a large deployment. Compare `used_memory` with `used_memory_rss`; a much larger RSS can be allocator fragmentation rather than retained keys, and a provider graph can spike while append-only persistence rewrites. Large `l1:live:*` keys point to live history volume. A Redis outage degrades the live view only; the inbox keeps accepting uploads and the workers keep committing them. The supplied Redis policy is `noeviction` so Channels group membership is never silently dropped; do not switch it to an eviction policy merely to suppress a memory alarm. Telemetry waiting to be processed is visible under **Agents & Settings > Telemetry processing** and per sensor on the Agents page. To inspect it directly: ```sql SELECT agent_id, count(*) AS pending, min(received_at) AS oldest, max(attempts) AS attempts FROM monitoring_telemetryinbox WHERE applied_at IS NULL GROUP BY agent_id ORDER BY oldest; ``` Rows with `applied_at` set are tombstones for sequences applied ahead of a sensor's contiguous prefix; they are deleted when the prefix passes them and pruned by `maintain_metrics` after two days once covered. A row that keeps failing for an unexpected reason is skipped after five attempts and recorded as a `telemetry_rejected` event carrying the sequence and the error. PostgreSQL uses memory for useful shared buffers, and a container-level graph can also include operating-system page cache, so a high graph alone does not establish a leak. Capture the provider's memory graph together with these snapshots during both normal and peak load: The web service runs Django under ASGI and therefore closes request database connections at the end of each request (`L1_DB_CONN_MAX_AGE=0`, the default). Each process reuses a small psycopg pool (`L1_DB_POOL=1`, default max 4, `min_size` 0) so short requests do not open a new PostgreSQL backend every time. Set `L1_DB_POOL=0` when an external pooler already owns connection sizing. Do not raise `L1_DB_CONN_MAX_AGE` on the supplied ASGI deployment; the pool is the reuse path, and persistent per-context connections can still accumulate idle backends there. Session `statement_timeout` (120s) and `idle_in_transaction_session_timeout` (60s) are applied unless the database URL already carries an `options=` query. ```sql SELECT name, setting, unit, source FROM pg_settings WHERE name IN ( 'shared_buffers', 'work_mem', 'maintenance_work_mem', 'autovacuum_work_mem', 'max_connections' ); SELECT state, count(*) FROM pg_stat_activity WHERE datname = current_database() GROUP BY state ORDER BY count(*) DESC; SELECT relname, pg_size_pretty(pg_total_relation_size(relid)) AS total_size, n_live_tup, n_dead_tup, last_autovacuum FROM pg_stat_user_tables ORDER BY pg_total_relation_size(relid) DESC LIMIT 20; SELECT temp_files, pg_size_pretty(temp_bytes) AS temp_written FROM pg_stat_database WHERE datname = current_database(); ``` Treat the last query as a cumulative counter and compare snapshots, rather than reading one absolute value. Size connection capacity against observed peak connections and the number of web, worker, and maintenance processes; add a provider-supported pooler before multiplying application replicas. `work_mem` can be consumed once per sort or hash node and parallel worker, while shared and maintenance memory have different lifetimes, so do not copy tuning values from another deployment. Use query plans and temporary-file growth to justify `work_mem` changes, and use sustained dead-tuple growth and autovacuum timing to justify vacuum tuning. Daily retention must remain healthy, but ordinary deletes and autovacuum make space reusable rather than immediately shrinking the database process or files; `VACUUM FULL` takes disruptive locks and is not a routine memory fix. Create the credential in Pulsar Operations as a super-admin: open `/console/admin-api/`, create a project with **Read VM inventory** scope, and create a token. Copy the one-time token into **Agents & Settings** here, test the connection, save it, explicitly map each Pulsar cluster to an L1 site, and run **Sync now**. Pulsar supports overlapping active tokens so the credential can be rotated without downtime. ### Destroy notices Addresses are recycled. When Pulsar destroys a VM and gives its IP address to the next customer within fifteen minutes, an import that has not run yet leaves the monitor holding the old guest's claim: the new owner's first packet opens a critical IP conflict against a machine that no longer exists, the destroyed guest is still listed on the inventory page, and reconciliation measures the new guest against the old one's expected addresses. The import interval is longer than the conflict window, so the import alone cannot prevent this. Pulsar therefore calls the monitor when it destroys a VM, and the monitor applies the destroy immediately: the expected VM and its interfaces are retired, the claims its MAC addresses hold on IP addresses are released, the devices carrying those MAC addresses are archived, and IP conflicts that only those claims were keeping open are resolved. Nothing is deleted -- traffic history, bindings and events stay as they were observed, because the period the VM ran is still a period somebody is billed for. A released claim is one with an end, and what ends is only its standing in decisions about the present. Open **Agents & Settings → Pulsar Operations & Aegis → Destroy notices**, press **Generate signing secret**, and copy the secret it shows -- it is shown once and the console cannot read it back. Configure it in Pulsar together with the URL the same panel prints, which is `https:///integrations/pulsar/vm-destroyed`. Generating a new secret refuses notices signed with the old one from that moment, so configure the replacement in Pulsar promptly. Pulsar POSTs JSON and signs it: ``` POST /integrations/pulsar/vm-destroyed X-Pulsar-Timestamp: 1789776000 X-Pulsar-Signature: sha256= {"cluster_id": 7, "destroyed_at": "2026-09-16T12:00:00Z", "virtual_machines": [{"id": 3456, "mac_addresses": ["aa:bb:cc:dd:ee:ff"]}]} ``` The signature is `HMAC-SHA256(secret, "." + raw body)`, hex encoded. `destroyed_at` and `mac_addresses` are optional; `mac_addresses` is what lets a guest created and destroyed between two imports be cleaned up, since the monitor never imported it and has no other way to know which MAC was its. Up to 100 virtual machines fit in one notice, so terminating an account is one call rather than fifty. The timestamp must be within five minutes of the monitor's clock, which is what stops a captured notice being replayed later against a rebuilt guest. The endpoint is unauthenticated apart from that signature, exactly like the Twilio callbacks, and reads nothing out of the body until it verifies. Configuring the secret is what turns notices on, so they are honoured even while **Enable automatic sync** is off: a signed destroy is the most current thing the monitor has about that guest, where a paused import is only an absence. To stop accepting them, generate a new secret and do not configure it in Pulsar. A notice is idempotent: redelivering it reports `duplicates` and changes nothing. A notice for a cluster this monitor does not import is acknowledged rather than refused, so Pulsar does not retry it forever. Notices are recorded as tombstones, which is also what stops an inventory page-through that began before the destroy from putting the VM back; `maintain_metrics` expires them on the ordinary retention window. Destroy notices are an optimization, not the only path. Every import still applies the same teardown to VMs billing no longer lists, and reports `destroyed_applied`, `devices_archived` and `destroyed_skipped` in its counts. A notice that is never sent, or is lost, is therefore applied at the next import instead -- which is the behaviour to expect before Pulsar is configured to send them at all. Traffic outranks a notice: a device that is observed again after being archived is un-archived and its claim made live, because a guest that is running is not a guest that is gone. Observations that predate the archive are late delivery and leave it standing. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.layeronecloud.com/network-monitor/operations/retention-and-memory.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.