quinzee.cloud

Development edition · 16 November 2026. Commands using repository paths, Make targets or sample projects require access to the Quinzee source checkout. No public installer is available. Examples use disposable local environments.

Administrator’s guide

13. Back up, recover and rehearse failure

A backup is useful only if the associated identities, configuration and service data can be restored together. Quinzee has several persistence boundaries. A copy of one convenient JSON file is not a complete environment backup, and a copy of a live SQLite main file may omit committed WAL content.

Step 1: inventory what must survive

Boundary Preserve
Cloud instance Complete data directory, bootstrap credentials, tenant/signing identity, module state, directory destination marker, runtime mailboxes/tombstones
PostgreSQL directory Consistent database backup plus the corresponding cloud identity/marker and private connection configuration
Application SQL/cache Engine-native consistent backups or stopped labelled volumes; record engine versions and helper ownership
Engine helper Private state, CA/signing/CRL metadata, credentials and ownership mapping
Hosting helper Private state, constrained CAs, registry and available immutable application images
TLS/DNS startup Configuration files, service certificates/keys, public trust distribution and private CA material under operator control
External observability Its separate persistent data, configuration, private auth/TLS files and retention choices

Decide the recovery point and downtime you can tolerate. This local platform has no built-in distributed backup coordinator, HA failover or point-in-time restore API. Use a maintenance window for a coordinated stopped-state snapshot. Application databases can use their normal engine backup tools, but consistency across database rows, blobs and queued work is the application's responsibility.

Step 2: take a consistent stopped-state backup

  1. Announce maintenance and stop producers, consumers and interactive changes.
  2. Stop hosted apps through their lifecycle APIs while the hosting helper is running. Wait for Stopped so queue triggers cannot keep mutating state.
  3. Take engine-native database backups or stop the managed engines cleanly. Stop the engine supervisor after reconciliation; account for its CRL service.
  4. Stop the LDAP frontend and cloud. Stop directory PostgreSQL cleanly if taking a cold filesystem backup instead of a PostgreSQL-native backup.
  5. Snapshot all listed state/configuration boundaries at that same maintenance point. Keep owner/mode metadata and encrypt/protect the backup as appropriate.
  6. Record checksums, image IDs, engine versions, instance paths and the date.
  7. Restart dependencies first, then cloud/helpers/apps, and verify a small create/read/update workflow before reopening access.

For a cloud-data archive only, after confirming the cloud and writers are stopped:

umask 077
mkdir -p .local/backups
tar --acls --xattrs -cpf .local/backups/cloud-data.tar \
  -C .local quinzee
sha256sum .local/backups/cloud-data.tar \
  > .local/backups/cloud-data.tar.sha256

That archive is one component, not the complete backup when PostgreSQL or external engine volumes are in use. It contains credentials and application data. Never attach it to a public issue or store it beside public web assets. Choose a new backup filename for each retained recovery point; the example filename is deliberately simple and would be overwritten on another run.

Step 3: restore into an isolated drill

Keep the original installation stopped while restoring its identity into another location. Running two copies of one tenant against the same PostgreSQL directory violates the writer lease. Restore the matching PostgreSQL snapshot, destination configuration/marker and cloud data as a set. Preserve helper ownership paths where possible; changing paths changes some helper ownership hashes. Do not manually rewrite ownership labels to adopt unknown containers.

Start the database dependency, cloud and helpers with an isolated network and unpublished/alternate host ports. Verify tenant ID, a known application identity, one shared group, one object per important service, a queued message and an application database row. Confirm private file permissions. For hosting, confirm the approved immutable image IDs still exist. Restart once more and repeat the checks. A successful extraction command alone is not a recovery test.

Restore procedures for external engines should use their own consistent dump or cold-volume method for the same version. Engine bootstrap/password/CA state must match the restored ownership. Quinzee does not currently provide an in-place resource import or volume adoption API. If a clean-room restore requires a new resource identity, plan an application-level data migration rather than editing private records to impersonate the old instance.

Step 4: respond to common recovery events

Directory database outage: stop administrative writes, recover PostgreSQL, then restart the cloud to reacquire its writer connection. Never remove the migration marker to revive stale JSON accounts.

Helper crash: restart the same helper with the same private state, mailbox root and network. Check current revision and heartbeat before restarting applications. Some containers may have continued running during the outage.

Interrupted deployment: inspect failed history, repair dependencies and redeploy from protected source/parameters. Completed resources can remain.

Lost TLS key or planned CA renewal: stop the affected endpoint, replace material through the certificate workflow, distribute the new public trust and restart. A fresh CA changes trust; it is not a transparent repair.

Accidental Delete: stop further cleanup/writes and assess backups. Resource and group deletion can remove data/volumes. Secret soft-delete recovery applies only to the supported secret workflow; it is not a universal undelete service.