Backup and restore

Daily backups of every environment

Sbarbase backs up every environment once a day, while it keeps serving, and keeps the last seven backups of each. Nothing needs to be set up. Each backup holds the environment's database (users, password hashes, tables, Storage metadata) and its own Storage files, with a digest of each. A failed backup sends a notification through the operator channels.

Set them in the unit's environment (systemd), or give them to deploy/compose.sh on every up (Docker, see settings).

Commands (with Docker, prefix docker compose exec sbarbase):

<environment> is the environment id from the console, or its runtime id (e_..., the path in its API URL). <scope> is that runtime id, storage or management. <backup> is the time shown by list, such as 20260924T030000Z.

Restore replaces one environment with its archived rows, users and files. It refuses before changing anything when the environment's Auth, REST or Realtime container runs an image other than the one this version pins, and never starts one on restored data. Its Auth, REST and configured Realtime services pause. Shared Storage also stops, so object access pauses for every environment until Storage restarts. PostgreSQL connection fences and bounded backend draining block admitted direct writers during replacement. The old database and objects stay aside until discard-previous removes them.

An interruption after replacement starts leaves a private operation journal. A handled error prints an exact recover-restore command. A hard kill may print nothing: after the process has stopped, run python3 lab/backup.py restore-status to obtain its exact recovery command without guessing the scope, backup or restore timestamp. Status reads private state without changing it or contacting Docker. Recovery verifies the archive and database identities. If the restored database has reopened, it preserves that database, new writes and new files; before rollback it also verifies the old object trees. Never rerun restore or remove the journal to bypass that decision.

This implementation is undergoing native interruption verification. Passing component or nominal restore checks does not establish the complete crash recovery contract. Restore admits PostgreSQL 17 and classified core background processes with supported hook libraries. It also admits the exact pinned Supabase image defaults when Cron uses ordinary client connections and the Cron/pg_net workers belong to the separate postgres maintenance database. The original authenticator supautils/safeupdate setting is checked by role and database identity. Changed preload settings, extension versions, scoped settings, subscriptions and unknown workers refuse before replacement; other native worker configurations remain unsupported. Archived owner and grantee roles must already exist. Database metadata comes from the archive, including ACLs, owner, comment, connection limit and database or role-specific settings, rather than the current live database.

If service start, endpoint publication or readiness fails after data replacement, the command exits nonzero and keeps a private atomic checkpoint in .lab/upstream/restore-completions/. It does not roll back the restored database or replay the archive. Follow the exact complete-restore command printed by the failure, including its lowercase restore timestamp, for example:

python3 lab/backup.py complete-restore e_<24-hex-digits> 20260924T030000Z 20261003t070000z
python3 lab/backup.py complete-restore storage 20260924T030000Z 20261003t070100z

Completion rechecks immutable archive hashes, native database container ownership/image, the restored database OID and the checkpoint identity. It retries service start, endpoint publication and the existing readiness gates, including shared Storage restart for a fenced environment restore. Writes committed after reopening remain intact; counts are not compared against the old backup. A successful retry durably records completion; repeating that same completion performs identity checks without restarting services. Environment readiness currently checks Auth HTTP 200 and shared Storage's listening TCP port. These gates do not prove REST, Realtime or Storage API health. Tests using service doubles do not establish actual Supabase readiness.

While any readiness checkpoint is pending or unverifiable, another restore, deletion of previous state and backup pruning refuse, including installation-run pruning. Create also refuses before writing a new backup. The current completed checkpoint's archive stays protected from pruning so its completion identity remains verifiable. Completed deletion/retention guards verify durable historical completion rather than requiring an old container to remain installed. An explicit completion retry still verifies current native database identity. A later successful restore replaces that scope checkpoint and releases the previous archive from this extra retention. Do not delete or edit checkpoint files to bypass a refusal, and do not rerun restore to resolve readiness failure. If restored database identity or archive identity changed, inspect and reconcile the current state before further recovery actions.

Current tests cover backup creation and restore data matching. Actual fenced interruption and recovery cases remain required before this restore workflow is accepted.

The installation manifest. Each daily run also writes .lab/backups/installation/<backup>/: the pinned images, the routing of every environment, the catalog's clients, projects, environments, memberships and jobs, the operator settings, and the names of the files in .secrets/. It holds no secret value: it is built from lists of allowed fields, and a webhook address is reduced to its host. It carries a digest and is kept as long as the environment backups. On a new server it tells you which pins and secrets the backups need.

Storage's shared metadata. Every create all run, the daily one and the one before each upgrade, also backs up storage_metadata, the one database Storage keeps for every environment: each environment's Storage registration, its signing keys and the migration state Storage records for it. It goes to .lab/backups/storage/<backup>/: a dump taken inside one snapshot while Storage serves, and its own manifest with a digest and the ids of the environments it registers, nothing secret. A failed dump fails the run. restore-storage <backup> replaces the whole database with it. Storage stops for every environment while it runs; the current database is kept aside until discard-previous storage, and an interruption follows the recovery instructions above, using storage as the scope. It refuses a backup that does not register an environment published now, because that environment's Storage would lose its registration. Restore it with the environment backups of the same run, environments first: the migration state it records has to match their databases (upgrades). It travels in the encrypted run set below; the per-environment copies of lab/offsite.py do not include it.

The management database. Every create all run also backs up management, the database of the management realm's Auth: the operator accounts, their sign-in factors and their sessions. An Auth release migrates it like the environments' Auth databases, so this backup is its way back. It goes to .lab/backups/management/<backup>/: a dump taken inside one snapshot while the management Auth serves, and its own manifest with a digest and the number of operator accounts and identities, nothing secret. A failed dump fails the run. restore-management <backup> replaces the whole database with it through the same fenced cutover as the other restores, and checks the operator account counts against the backup before anything is renamed. The management Auth and the shared Storage process stop while it runs, so operators cannot sign in and every environment's Storage pauses; nothing else does. The current database is kept aside until discard-previous management, and an interruption follows the recovery instructions above, using management as the scope. Operator accounts, passwords, factors and sessions changed since the backup go back to what they were then. The console reads the management Auth's address only when it starts: when the restore brings the management Auth back on a new address, restore-management, complete-restore, recover-restore and restore-status say so until Sbarbase restarts, and Sbarbase must be restarted before operators can sign in again. A restart within about ten seconds of the move cannot be told apart from a console that started before it, so the notice may stay after it; restart once more and it clears. It travels in the encrypted run set below when that run also backed up at least one environment; a run with none is not copied off the host, so on an installation with no environment yet the operator accounts stay on this host only. The per-environment copies of lab/offsite.py do not include it.

There are two independent ways to copy backups off the server, described below. Configure one of them. The encrypted run sets (the second) are recommended: each set carries the environment backups, Storage's shared metadata, the management database and the installation manifest, which a new server needs after the old one is lost, and a restore served only from a run set in Cloudflare R2 passed on the real server on 2026-10-17.

Copies off the server

Backups are written to .lab/backups/ on this server. To survive losing the server, let Sbarbase copy each new backup, encrypted, to S3-compatible storage: Cloudflare R2 (10 GB free), Backblaze B2, AWS S3, Wasabi or your own MinIO.

1. Create a bucket and an access key that can read, write, list and delete in it.
2. Give Sbarbase the settings once. They are read from stdin, never from the command line, and kept in .secrets/offsite.json:

python3 lab/offsite.py configure <<'JSON'
{"endpoint": "https://<account>.r2.cloudflarestorage.com", "bucket": "my-backups", "region": "auto",
 "access_key_id": "…", "secret_access_key": "…", "passphrase": "a long phrase only you know", "keep": 30}
JSON

It proves the settings by writing, reading and deleting a test object. From then on the daily backup copies each new backup by itself and keeps the newest 30 per environment in the bucket; a failed copy is reported like a failed backup.

Every file is encrypted on this server before it leaves (AES-256-GCM, the key derived from your passphrase), so the storage provider never sees your data. Keep the passphrase somewhere other than this server: without it the copies cannot be read, by you or anyone.

A fetched backup is checked against its manifest and then restored with backup.py restore as usual. An upgrade's own safety backup stays local (backup.py create all --local-only), so storage that cannot be reached never blocks an upgrade.

CI runs this cycle on a clean machine with every change against a throwaway MinIO: a backup copied by itself, only ciphertext in the bucket, a wrong passphrase refused, the copy fetched and restored (evidence).

Encrypted run sets off the server

Backups are written to .lab/backups/ on this server. To keep a copy elsewhere, point Sbarbase at one S3-compatible bucket (Amazon S3, Cloudflare R2, Backblaze B2, MinIO and others). After each daily run, the run's backups and its installation manifest are packed into one file, encrypted with AES-256-GCM, and uploaded as <prefix><backup>.sbb. The target keeps as many sets as SBARBASE_BACKUP_KEEP; older sets under the same prefix are deleted, and other objects are never touched.

1. Create the encryption key. Sbarbase never creates one on its own:

   python3 lab/backup.py offsite-key .secrets/upstream/offsite-key.json

   Copy

   The file is mode 0600. Keep a copy of it away from this server: without it no copy can be decrypted, and anyone with it and the bucket can read every backup.

2. Put the bucket's access key in a 0600 file, for example .secrets/upstream/offsite-s3.json:

   {"schema": 1, "accessKeyId": "<access key id>", "secretAccessKey": "<secret access key>"}

   Copy

   Give that key only list, read, write and delete rights on the bucket.

3. Write .lab/upstream/backup-offsite.json:

   {
     "schema": 1,
     "s3": {"endpoint": "https://s3.eu-central-1.amazonaws.com", "region": "eu-central-1",
            "bucket": "example-backups", "prefix": "sbarbase/",
            "credentialsFile": ".secrets/upstream/offsite-s3.json"},
     "keyFile": ".secrets/upstream/offsite-key.json"
   }

   Copy

   The endpoint must use https. Paths are relative to the checkout.

The next daily run copies itself. A copy that fails (the bucket is unreachable, the key file is missing or readable by others) sends a backup.failed notification and prints the reason; the local backups are never changed or removed, and nothing is deleted on the target. The run can then report both a completed backup and a failed copy.

Only the daily run (create all) is copied; a backup of one environment taken by hand stays on this server.

Restore from the target, on this server or on a new one with the same environment published. A new server needs the three files above first: the configuration, the credentials file and your copy of the key.

python3 lab/backup.py offsite-list
python3 lab/backup.py restore <environment> <backup> --offsite

The set is downloaded, its authentication tag is checked before anything is unpacked, and each backup is checked against its digests before restore uses it. A backup already on this server is kept as it is.

Restore on a new installation

A backup restores into an environment that exists and is published. On a new server, after losing the old one, that environment does not exist yet. Each backup records the ids of its client, project and environment (ownership in manifest.json, since 2026-09-23), so the new installation can recreate them exactly and then restore into them:

1. Install Sbarbase on the new server (operator setup) at the same release, so the pinned images match; the installation manifest lists them.
2. Put the backup under .lab/backups/<runtime>/<backup>/: copy the directory from wherever you keep it, or fetch it from the off-host target as described above.
3. Run the three steps below as an installation operator. relink recreates the client, project and environment with their original ids and runtime id, and the worker provisions an empty environment of that name. When environments shows it succeeded, restore fills it from the backup.

sbarbase relink <runtime> <backup>
sbarbase environments
sbarbase restore <runtime> <backup>

relink is careful on purpose. It never attaches a backup to a client or project by name: when a client with the recorded name but another id exists here, it refuses, and so it does when the recorded project or environment id already means something else here, or the runtime id belongs to another environment or to one deleted on this installation. A refusal changes nothing. You become the owner of a client it creates; if the recorded client already exists here, you must already be one of its owners. Asking again after a success changes nothing.

What a backup does not carry, and what you do after the restore:

• Members. Account ids belong to the old installation's sign-in, so nobody is added. Invite people again.
• API keys and access tokens. Issue new publishable and secret keys and update your apps; agents need new sbp_ access tokens.
• Vault secrets. An environment backup is a database dump and its files; it does not include the server's Vault root key, so the secrets in it cannot be read on a new server (database access). Store them again there.
• The JWT signing key. The new environment has its own. Users sign in again with their old passwords, which the backup keeps; existing sessions and signed Storage URLs stop working.
• Direct database access, Studio, Realtime and Edge Functions. Turn them on again where you used them; database access gets a new password.

This path is covered by unit tests of the catalog, the API and the command, and it was rehearsed end to end on 2026-09-25: two backups taken in one VM while both environments served traffic were restored onto a second, freshly installed VM with these three commands, and a user from each old environment signed in with their old password through a new key (evidence). Those environments had never used Studio, Realtime or direct database access. One risk stays untested: such an environment has logins for those features that its database grants access to, a new installation does not have them until the feature is turned on, and the restore may refuse. Turn the feature on for the new environment before you restore into it.

Other manual backups

The copies below are older manual procedures that need supervision. For the design behind them, read recovery.

Whole-server cold backup

The simplest backup is a cold copy of everything, taken with all services stopped:

1. Stop everything: the supervisor (systemctl stop sbarbase.service on a server, Ctrl+C for lab/dev.py), a runtime started by hand (/usr/bin/python3 lab/durable_runtime.py stop) and, if an environment has been moved to a recovery target, that target (/usr/bin/python3 lab/target_runtime.py stop).
2. Confirm with docker ps that no sbarbase-durable-* and no sbarbase-restore-* container is running.
3. Copy the Docker volumes of both the sbarbase-durable-* containers and the sbarbase-restore-* containers (find them with docker inspect), plus the checkout's private state directories .lab/ and .secrets/. A moved environment's data lives only on the sbarbase-restore-* volumes, so copying the durable volumes alone misses it. Keep the copy encrypted and off the server; .secrets/ holds every generated credential.
4. Start the supervisor again.

This copies the whole server at once. Restoring it restores every environment to that moment; it cannot restore one environment alone. This procedure has not been rehearsed as a restore on a clean host.

One environment: encrypted export

lab/recovery-export.py writes an encrypted bundle of one environment: its database dump, scoped logins, grants and settings, files with extended attributes, Storage tenant configuration and URL-signing keys. Start the owned runtime first (/usr/bin/python3 lab/durable_runtime.py up), then run /usr/bin/python3 lab/recovery-export.py.

Exporting one environment currently takes every environment on that engine offline. To get a consistent snapshot, the export stops Auth, REST, the management Auth realm and the shared Storage process for the whole source placement, not only for the environment being exported, and it leaves them stopped afterwards. Every app on the server and the console login are unavailable until you start the runtime again. Plan it as a maintenance window. The ciphertext and its separately stored key are written under .lab/upstream/ with mode 0600, and .lab/upstream/recovery-latest.json points to them.

Be aware:

• It currently selects the first ready environment of the durable lab fixture; it is not yet a general "export environment X" command.
• The bundle is held in memory and capped at 32 MiB.
• lab/cutover-export.py is the fenced variant used when an environment is being moved. It puts the environment in maintenance, disables its service logins and closes its source database afterwards. It leaves the source fenced on purpose, and reopening it is a deliberate reconciliation step. Do not use it as a routine backup command.

Restore into a separate engine

The restore path is a chain of scripts that were built against the lab fixture:

/usr/bin/python3 lab/recovery-restore-db.py          # fresh pinned engine, transactional restore, verification
/usr/bin/python3 lab/recovery-check-services.py      # original Auth and REST against the restored database
/usr/bin/python3 lab/recovery-check-storage.py       # Storage metadata, files and signing keys

The target descriptor is private (.lab/upstream/recovery-target.json). Moving traffic to the restored copy is done through the routing record with lab/target_runtime.py; see target lifecycle.

When Sbarbase cannot start after a way back

A Storage release migrates the shared storage_metadata. If the previous Storage image cannot start on the migrated database, the way back leaves the previous version failing to start, and its service manager keeps restarting it (systemd every 10 seconds, Docker's restart: unless-stopped). Every restore needs the database running. So stop Sbarbase first: sudo systemctl stop sbarbase, or docker compose stop sbarbase. Then /usr/bin/python3 lab/backup.py start-database starts the database container alone. Run it and the restores below as the service account, which owns the state: under another account it refuses, because the private state is not that account's own (under sudo, the generation pin). It refuses while Sbarbase's supervisor holds its lock, but a failing service holds it only while it tries to start: between restarts the lock is free, the check passes, and the restores do not check it again, so a restart in the middle would stop the database under a restore and start the runtime again against the unfinished restore; nothing at startup checks for one. The restore then stops, and what it prints depends on where it was hit: a plain refusal, recover-restore or complete-restore. That is why the service must be stopped, not only failing. It also refuses while a generation migration waits, and starts only the container the HBA generation pin names, on the image this version's lock pins, then waits until it accepts connections. Nothing else starts. The way back lands on the previous version, so the command is there only when that version has it. Under Docker, with the container stopped, run these commands with docker compose run --rm --entrypoint /usr/bin/python3 sbarbase lab/backup.py ... instead of docker compose exec; this form has not been tried.

Then restore what the release migrated from the backups taken before the upgrade, storage_metadata first: restore-storage <backup>, then restore <environment> <backup> for every environment. The order matters here, unlike in the undo after a working way back: every restore finishes by starting Storage again, and Storage starts only once storage_metadata is the one it can run on, so an environment restore taken first cannot complete, and its pending completion refuses restore-storage, every other restore and discard-previous; complete-restore and recover-restore fail the same way while Storage cannot start. No command gets past that yet, so keep this order. Sbarbase is stopped meanwhile, so nothing reaches Storage between the restores. Then start Sbarbase. This works when the containers that are left run the images the current version pins: a restore refuses a Storage container on another image, and an environment restore refuses, before it changes anything, when that environment's Auth, REST or Realtime container runs another image. That holds when the upgrade you are undoing changed Storage alone. Each Sbarbase version changes one upstream component, but one upgrade can cross several versions: the update channel offers the newest reachable release, and lab/upgrade.py start --to takes any version. An upgrade that also crossed an Auth, REST or Realtime change leaves those containers on the newer images, since the way back stops before it replaces them, and every environment restore then refuses; that case is not covered here. To keep a Storage release undoable this way, upgrade to the version just before it first. It is unit tested only: no way back has failed this way on a live host yet.

If a restore is interrupted

A failed restore leaves its descriptor behind, and a plain rerun refuses. Do not edit that state by hand:

/usr/bin/python3 lab/recovery_reconcile.py                     # stop the retained target containers
/usr/bin/python3 lab/retire_recovery_target.py --reason "why"  # archive the descriptor, keep containers and volumes
/usr/bin/python3 lab/recovery-restore-db.py                    # a fresh restore may now run

Limits

• The daily backup, its retention and the off-host copy are described at the top of this page. The limits in this list apply to it too: no point-in-time recovery.
• The cold backup and the encrypted export above mean downtime for all environments on the engine, not only the one being saved.
• The off-host copy speaks S3 only; SSH or rsync targets are not built. The encrypted run sets were restored from Cloudflare R2 on the real server (one environment, no application data, no interruption); lab/offsite.py is tested against a local MinIO only. Restoring onto a new server from the off-site copy has not been exercised.
• One set is one upload, so a set larger than the provider's single upload limit (5 GiB on Amazon S3) fails. The set is written encrypted to this server before the upload, so the disk needs room for it.
• Retention on the target is by count, not by age.
• An encrypted run set written since 2026-09-26 carries the storage/ folder, and a version older than that refuses the whole set as holding an unexpected entry. Sets written since 2026-10-15 also carry the management/ folder, which every earlier version refuses the same way. After a way back to such a version, offsite-fetch and restore --offsite fail for those sets; the local backups, including those taken before an upgrade, are unaffected.
• Restoring needs the environment published on that server. On a new installation, sbarbase relink creates it first (above); a backup taken before 2026-09-23 records no ownership and must be restored into an environment created by hand. Database roles shared by the whole engine are not in the set.
• Restore has been rehearsed on one host with a test fixture, not on a server with real client data. The storage_metadata backup and restore-storage have unit tests only; neither has run against a live Storage yet. The same holds for the management backup and restore-management, which have not run against a live management Auth yet.
• Once a restored target has accepted writes, going back to the old source is unsafe without reconciliation.

The full record, with check counts, is independent restore.

Private local installation collection

sbarbase backup collect-installation --output /absolute/private/directory collects the original complete Catalog and managed key store, every file in .secrets/upstream, top-level JSON state/configuration, retained Functions code including version directories and shared imports, and present image lock files. The destination must be absent, outside the installation, and under an owned private directory. Files are mode 0600 and directories are private. Keep this plaintext collection private. It includes secrets and is unencrypted.

The command reports local-collection-only, complete: false, encrypted: false, and restore_admitted: false. Its collection.json records material digests privately and identifies the missing native suppliers. This collection cannot be used as a complete backup, a fresh-host restore package, or a recovery receipt for deletion. It does not invalidate preserved execution leases, MFA grants or credentials, and does not activate any restored store.

Individual Catalog and KeyStore read transactions retain consistent original SQLite snapshots. Changed configuration/file identity or tree membership refuses publication. This does not establish a transaction across those stores or quiescence of native database and object writers.

sbarbase backup installation --output /absolute/private/file --key-file /absolute/private/key is reserved for complete authenticated packaging. It currently refuses because genuine native feature inventory, Storage signing material provenance, source host/resource admission, and native database/object suppliers are not connected. Complete distinct-host recovery with source state inaccessible remains required. Existing backup now, offsite copies and environment restore retain their current scope.

The collector uses the original supported Functions entrypoint set: index.ts, index.js, index.tsx, index.jsx and index.mjs. The pinned TypeScript compiler parses captured modules to check static local imports and re-exports, including shared imports, against the exact deployed version tree. Missing local modules and unsupported dynamic module expressions refuse local collection. Remote npm, jsr, HTTPS and Node module references are recorded privately as unverified external dependencies. Deno execution, arbitrary runtime code loading and external module availability require the later native recovery checks. Directory enumeration errors refuse collection; an unreadable private subtree can never be silently omitted. Directory identity/membership changes also refuse.

Local collection requires Node.js 22.6 or newer and the pinned TypeScript 7.0.2 package installed from the project's frozen lockfile. The helper uses standard JavaScript modules and does not require Node TypeScript stripping support. The compiler API uses Node pipe handles and cannot run under Bun. Node version and parser imports preflight before source material is read or a collection is staged. Missing or unsupported tools refuse explicitly. The parser subprocess and its compiler descendants have a finite shared deadline and are cleaned up together on failure.

The supported control-plane container installs Node from the same immutable Ubuntu package snapshot as Python and verifies the minimum version during its build. The source CI jobs declare Node 22.23.2 explicitly. These runtime changes require their actual container/CI checks before production activation; the local source suite does not establish that a rebuilt container was deployed.

Edit on GitHub