# Server deployment

Current published checkpoint `e08ed3b2684bdf04f0288c7162747a2783177026`: [CI 37600158231](https://github.com/M7MMAD-OMAR/sbarbase/actions/runs/37600158231) passed all four jobs: general checks, Python compatibility, Docker installation and empty-host acceptance. Python 3.12 passed 2102 tests. [Website 37600311118](https://github.com/M7MMAD-OMAR/sbarbase/actions/runs/37600311118) passed build and tests. Empty-host acceptance now runs automatically on main pushes as well as explicit manual requests; pull-request behavior is unchanged.

The [current native checkpoint summary](https://github.com/M7MMAD-OMAR/sbarbase/blob/main/docs/evidence/ci-e08ed3b-2026-10-07/summary.json) binds the new Docker artifact to 13 reports and 254 passing checks for project creation, Supabase features, backup/restore, upgrades, rollback and restart. This is the new run's fresh-only artifact, not the earlier baafbad archive or historical installation transcription. Its runtime evidence is bound to core code commit `e08ed3b2684bdf04f0288c7162747a2783177026`; a subsequent documentation publication preserves that identity and does not establish runtime acceptance for later code changes.

The [fresh empty-host evidence](https://github.com/M7MMAD-OMAR/sbarbase/blob/main/docs/evidence/ci-e08ed3b-2026-10-07/empty-host/summary.json) contains six reports and one run manifest, with 58 unique checklist checks and a separately reported 21-check live Auth bootstrap. The identical rehearsal handoff copy adds no checks. The native log and final script guards establish that the shipped system unit became active and its console answered after the rehearsal. The earlier JSON unit snapshot records the deliberate inactive rehearsal phase; it is not the final restored service state. This is a fresh Ubuntu runner installation checkpoint, not physical reboot, Kali verification, complete lost-source recovery, cold/warm Cron continuity or production acceptance. Static-serving and TLS stub checks retain their separate scope; a public certificate is not proven.

A separate [Fedora 44 systemd descriptor probe](https://github.com/M7MMAD-OMAR/sbarbase/blob/main/docs/evidence/systemd-scope-fd-probe-2026-10-07/summary.json) preserved five descriptor identities across same-PID Python exec under bounded systemd scope custody. Its two-file public archive contains the probe and summary. It launched no guest or Docker container and accepts only that Python descriptor behavior, not a Linux guest, reboot or portable production deployment.

Historical integration checkpoint before e08ed3b, 2026-10-07: Docker installation CI passed after `baafbad`, with 2086 Python tests and 254 current native checks. Manual standalone-service acceptance failed at the socket configuration handoff; its later rerun passed on e08ed3b. Full recovery and production remain unaccepted. See [current status](https://base.sbarah.com/en/docs/reference/status/); the following checkpoint paragraphs preserve historical snapshots and their limits.

Historical checkpoint, 2026-10-06: complete snapshot verification passed 1449 tests with no failures, errors, warnings or skips and independent actual review. Repairs cover SQLite and HTTPError response closure, tempfile ownership and expected test diagnostics. All 52 preparation cases also passed; native PostgreSQL startup, Cron/Vault restart continuity and physical restore remain unaccepted. See [current status](https://base.sbarah.com/en/docs/reference/status/) for CI results and evidence scope.

The separate public55 V3 role has actual MET: 25 component and 30 pump tests passed, original 290 and added 4 actual mode/byte captures matched, and all three known helpers were removed with no unresolved resources. Public BusyBox V8 help metadata also has actual MET: fixed-path Bash 5.3.3 and BusyBox 1.37.0 help returned zero, with its one helper removed. Neither result accepts private request operation, timeout timing/signal behavior, installed binary source provenance, candidate startup, cold/warm Vault or Cron continuity, physical restore or production.

The preserved warning-window V4 negative run reported 1447 ordinary tests OK but six ResourceWarnings, native 1 and observed window 839 for every warning. Three records have exact sqlite3.Connection type; three report tempfile.py line 484 with NoneType source. Observation windows are not allocation causes, and the three tempfile origins remain unknown. At that historical checkpoint, fresh complete source/identity/mode/full bindings, final-revision CI and manually dispatched empty-host acceptance remained required; published 0e745394 CI and Website results retain only their earlier revision scope.

Runbook for the native Linux installation path. The primary container workflow is described in [Install with Docker](https://base.sbarah.com/en/docs/guides/docker/). September workstation and local-VM rehearsals below retain their original scope; they are not fresh production acceptance.

A real server runs main at sbarbase.sbarah.com (Ubuntu 26.04, systemd, the TLS proxy behind Cloudflare), upgraded from 0.2.0 on 2026-10-16; local and R2 restores passed there on one environment without application data. Historical published `0e745394` passed Docker installation CI, while empty-host acceptance was skipped at that checkpoint. No independent public-server pilot is established. Current startup tooling defines only an explicit prepare-only material-handoff status; no actual native material-preparation run is accepted by this guide. The earlier V9 initV2 run passed 52 focused tests but its historical actual 1447-case full regression failed with one failure and 15 errors.

Earlier mode, scratch and warning verification attempts remain historical refusals bound to their own source and date. Resource ownership and fixture diagnostics were repaired, and complete verification of that earlier snapshot passed 1449 tests. This does not transfer native startup or recovery evidence from another run. The [status history](https://base.sbarah.com/en/docs/reference/status/) preserves the earlier observations and their limits.

## Prerequisites

| Requirement | Why |
|---|---|
| Linux x86-64 host with Docker (native daemon, not remote) | every placement runs pinned containers |
| Bun on PATH (for a system service, add its directory to the unit's `PATH`, e.g. `/home/sbarbase/.bun/bin`) | package manager, console build, gateway checks |
| `/usr/bin/python3` 3.12 or newer, with `cryptography` | the lab runtime uses f-strings that need 3.12; `cryptography` encrypts recovery bundles and off-site backups |
| Git checkout of this repository | state and lock files live in the checkout by default |
| Headroom: on an empty server 4352 MiB available (1792 MiB for the database, Storage and management Auth containers plus a 2560 MiB reserve) and at least 2 CPU cores; more as environments are added. The preflight states the exact figure and refuses below it | The requirement is derived from the placement the next start runs: on an empty host the three system containers at their tier limits, afterwards every retained container at its own limits (each environment adds 512 MiB and 0.5 CPU of ceilings for its Auth and REST), plus the reserve, plus, on an installation that has been moved, the measured cost of the running source stage (`docs/evidence/source-stage-footprint.json`). CPU ceilings may add up to twice the cores after one core is kept for the host; the cgroup pressure gate refuses new work under real contention. The development host's retained split placement still needs 5888 MiB of limits plus the reserve. The preflight prints the composition, so a refusal names each term |
| A service account that exists, holding the checkout | the unit runs as that account (`User=`), so `--apply` refuses an account that does not exist instead of installing a unit that cannot start. The shipped default is `sbarbase`; name the server's account with `--service-user`, `--home` and `--bun-dir` (also forwarded by `deploy/server-acceptance.sh`) |
| Docker socket access for the service user | Admission pins the declared local Unix socket, default `/var/run/docker.sock`. Docker contexts, remote endpoints and Desktop are outside this candidate. Nondefault socket and data root must be declared consistently for the service and acceptance run; see the [host contract](https://github.com/M7MMAD-OMAR/sbarbase/blob/main/docs/engineering/HOST-PREFLIGHT.md) |

Pinned images are pulled by digest on install; no floating tags are used. See
[upstream update policy](https://github.com/M7MMAD-OMAR/sbarbase/blob/main/docs/engineering/UPSTREAM-UPDATE-POLICY.md) before changing any pin.

## Install

One command on the server covers prerequisites, preflight, the full rehearsal and
the acceptance evidence:

```
deploy/server-acceptance.sh --rehearse --bootstrap-file /path/to/operator.json
```

Without `--rehearse` the same command runs the prerequisites and the preflight
only. It refuses before touching anything when a prerequisite is missing (docker, bun,
git, `/usr/bin/python3` 3.12+, native Linux daemon), when the bootstrap file is
not mode 600, or when the preflight reports a blocker, and it never prints a
secret. The step-by-step sequence below is what it runs, for an operator who
wants to drive each stage by hand.

```
/usr/bin/python3 lab/install_server.py check        # read-only preflight, non-zero on blockers
/usr/bin/python3 lab/install_server.py plan         # print the exact steps
/usr/bin/python3 lab/operator_file.py /root/sbarbase-operator.json   # prompts, no echo, mode 0600
/usr/bin/python3 lab/install_server.py install --bootstrap-file /root/sbarbase-operator.json
```

Create the file with `lab/operator_file.py` rather than by hand: it prompts for
the email, the organization and the password twice without echoing, refuses a
password shorter than 12 characters, a malformed email, a relative path, a
symlink and an existing file (unless `--force`), creates it with mode 0600, and
never prints the password. For automation it also accepts the same JSON on
bounded stdin with `--stdin`, so a password never reaches an argument or the
shell history.

`install` performs, in order: preflight, private state and secret directories
(0700), pinned image pull when not local, `bun install` when needed, console
build, owned runtime startup, and the operator identity bootstrap. The
bootstrap file is a 0600 JSON object with exactly `email`, `password` and
`organization`; it is piped on stdin and never passed as an argument or printed.

A fresh install initializes exactly one HBA generation for the new database. A
retained installation (containers already present) is refused until its source
and current recovery target carry generation pins:

```
/usr/bin/python3 lab/adopt-retained.py source
/usr/bin/python3 lab/verify_retained.py source
/usr/bin/python3 lab/adopt-retained.py target
/usr/bin/python3 lab/verify_retained.py target
```

Adoption starts and stops only the captured database container and preserves its
existing rules byte for byte. Never delete a pin, journal or receipt to bypass
the gate.

## Supervise

Install and start the unit with one command. It renders the shipped unit for this
installation (paths, service user, Bun directory), verifies the result with
`systemd-analyze verify`, then installs, reloads, enables and starts it:

```
sudo /usr/bin/python3 lab/install_server.py supervise --apply \
     --service-user ops-account --home /srv/ops-account --bun-dir /srv/ops-account/.bun/bin
```

`sudo` replaces `PATH`, so a Bun outside a system directory must be named with
`--bun-dir` or the unit starts without it.

Without `--apply` it only renders and verifies, prints the exact commands it would
run, and writes `docs/evidence/supervisor-unit.json`. It refuses to install a unit
that does not verify, refuses `--apply` without root, and refuses to install for an
account that does not exist on the host. Point it at a different layout with
`--service-user`, `--home` and `--bun-dir`; the shipped unit is never hand-edited.
`deploy/server-acceptance.sh` forwards the same three flags, so the one-command
acceptance path can name the server's account too.

The unit carries what updates need ([upgrades](https://base.sbarah.com/en/docs/guides/upgrades/)):

- Its first `ExecStartPre` runs `deploy/host-preflight.sh --runtime`. Read-only host
  admission refuses before locks, upgrade changes or leftover runtime cleanup on every restart.
- The next `ExecStartPre` is the upgrade guard (`lab/upgrade_guard.py`, or the copy
  an upgrade left in `.lab/upgrades/guard.py`). It runs before the preflight and
  before any code of the version the checkout holds, and moves the checkout back
  when a new version keeps failing its start.
- After the guard, `ExecStartPre` (`lab/leftover_runtime.py`) handles a supervisor that
  was killed (SIGKILL, the OOM killer) instead of stopping. Docker, not the unit,
  owns the containers, so they keep running, and the preflight used to refuse every
  later start because owned containers were running. This step stops them the way
  the supervisor's own stop does (`docker stop`, every container and volume kept),
  but only when no supervisor, worker or effect owner holds its lock and no
  provisioning receipt, HBA journal or migration record waits for reconciliation.
  Otherwise it refuses and names the reason. The supervisor runs the same step
  after it takes its locks, which covers Docker and terminal starts.
- `RestartForceExitStatus=42`: after an update or rollback from the console moves
  the checkout, the supervisor stops everything cleanly and exits with code 42 so
  systemd starts it again on the new version. `Restart=on-failure` already covers
  that exit; the line keeps it so if the restart policy changes.
- `StartLimitIntervalSec=0`, so systemd never gives up restarting while the guard
  needs several starts to go back, and `TimeoutStartSec=600`, because the guard's
  way back reinstalls dependencies before the preflight runs.

A unit installed before host admission was added must be rendered and reinstalled
with `supervise --apply` to obtain the admission boundary on every restart.
Without the leftover cleanup line, a start after an
unclean stop still refuses at the preflight; stop the leftover containers once with
`/usr/bin/python3 lab/leftover_runtime.py` as the service account. Reinstall the
unit with `supervise --apply` when you move to the version that has them.

Four things must be true before the unit can serve, and the preflight names each
one rather than failing obscurely:

1. **The service account exists.** Create it and give it the checkout, or install
   with `--service-user`/`--home`/`--bun-dir`.
2. **The service can reach the declared local Docker daemon.** The default is
   `/var/run/docker.sock`, with socket permission for the service account.
   `DOCKER_CONTEXT` is refused. For a nondefault local socket or daemon data root,
   set `SBARBASE_DOCKER_SOCKET` and `SBARBASE_DOCKER_DATA_ROOT` in the service's
   drop-in and acceptance environment. Any `DOCKER_HOST` must match that socket.
   A drop-in uses `Environment=SBARBASE_DOCKER_SOCKET=/run/docker-custom.sock`,
   `Environment=DOCKER_HOST=unix:///run/docker-custom.sock` and
   `Environment=SBARBASE_DOCKER_DATA_ROOT=/srv/docker` with the actual existing paths.
   The first admission step refuses missing or mismatched paths without creating them.
   When Docker is unreachable, preflight reports the endpoint and does not guess
   about pinned images or existing containers.
3. **The host has the memory.** The next start needs the limits of the
   containers it runs plus a 2560 MiB reserve: 4352 MiB on an empty server, more
   with each environment, 8.8 GiB for the development host's retained split
   placement. The unit's `ExecStartPre`, the preflight and the runtime admission
   derive the same figure, and the gate refuses with `host_memory_headroom`
   rather than half-starting.
4. **Write access stays inside the checkout.** `ReadWritePaths` names the
   installation root only. A `ReadWritePaths` entry for a directory that does not
   exist makes systemd fail the unit with `226/NAMESPACE` before it runs anything,
   so the unit never lists a path the installation does not create; every secret
   lives in `<checkout>/.secrets/upstream`. The rendering has been verified on the development host and the
install commands are recorded in that evidence file; the install itself needs root
on the target server.

The unit runs `lab/dev.py`, which builds the console, starts the owned runtime,
runs the API and the provisioning worker, and stops the runtime on SIGTERM.
`ExecStartPre` re-runs the preflight, so an unfit host fails before any
container is touched. For a foreground run instead, execute
`/usr/bin/python3 lab/dev.py` in a terminal and stop it with Ctrl+C.

The one command that produces the acceptance evidence is
`deploy/server-acceptance.sh` (below); it runs the preflight, the checks and the
rehearsal in order and keeps the handoff copy. To drive the two stages yourself
instead:

```
/usr/bin/python3 lab/install_server.py check
/usr/bin/python3 lab/deployment_rehearsal.py --bootstrap-file /path/to/operator.json
```

The install itself needs the placement's memory plus the reserve (4352 MiB on an
empty server): the owned runtime refuses to start when the host cannot support the
placement containers plus the reserve, and names the reason (`host_memory_headroom`) rather
than half-starting. The installer releases the installation operation lock after
the console build and before it starts the owned runtime, because the runtime
takes that lock itself and holds it for its lifetime.

On a server where the unit still has to be installed, run the acceptance path as
root with `--install-unit`: it renders and verifies the unit, installs and starts
it, proves the console and the TLS termination, releases the unit so the rehearsal
can own the containers and state, runs the rehearsal with the unit required,
starts the unit again and asserts it is active, and leaves the evidence in one
place. systemd calls the unit active the moment the supervisor process starts,
long before the console can answer, so after every start of the unit the run
waits until the console answers over loopback, at most 300 seconds
(`--console-timeout` changes the bound), and fails with the last thing it saw
when it does not. The release happens whenever the unit is found active, with or without
`--install-unit`, and a trap starts it again on any exit, so a failure in the
middle of the run cannot leave the installation down. Two supervisors cannot own
the same containers, so the rehearsal never runs against a live installation.

```
sudo deploy/server-acceptance.sh --rehearse --install-unit \
     --service-user ops-account --home /srv/ops-account --bun-dir /srv/ops-account/.bun/bin \
     --bootstrap-file /path/to/operator.json
```

`sudo` replaces `PATH` with a secure default, so a Bun installed under the
invoking user's home is invisible to the script and to the unit it installs.
Pass `--bun-dir` (the script adds it to `PATH` and uses it in the unit) whenever
Bun is not in a system directory; without it the script stops at the
prerequisite step and names the flag.

`sudo` is used for the two unit steps only. Everything that touches the
installation runs as the account that owns it (`--service-user`, or the checkout's
owner when the flag is omitted), because the state carries its owner and the
runtime refuses a process whose uid does not match the files it holds
(`HBA ownership inode mismatch`): running the preflight, the checks or the
rehearsal as root against a service-account installation fails, and a root-run
install would leave files the service cannot use. The script does that
substitution itself, so the single `sudo` invocation above is still the whole
command.

Admission selects the declared local socket explicitly and forwards the declared
profile, socket and data root when a step runs as the installation account. Saved
Docker contexts do not select the endpoint. `sudo` can discard shell exports, so
supply the same public deployment inputs explicitly to the root invocation and
service drop-in. The default endpoint can also be restated:

```
deploy/server-acceptance.sh --rehearse --docker-host unix:///var/run/docker.sock
```

For a custom local endpoint, declare its existing socket and actual Docker data
root, then pass the matching `--docker-host`. A mismatch or context override
refuses before rehearsal effects. The native route still requires host Python,
Bun and systemd; the primary [Docker route](https://base.sbarah.com/en/docs/guides/docker/) uses the public launcher.

An acceptance run writes its rehearsal to
`docs/evidence/server-acceptance-rehearsal.json` and copies it to
`server-acceptance-latest.json`, leaving the plain
`docs/evidence/deployment-rehearsal.json` untouched. That record carries the host
facts (kernel, Docker, Bun, Python, headroom, free disk), the exact command that
ran, the full pin set with digests, the state of the `sbarbase.service` unit,
start and finish times, and one row per check with its result. A rehearsal that
cannot run records the blocking finding instead of a pass, and exits non-zero.

### Check: does the supervised path work, not just a direct run?

The rehearsal runs the supervisor directly. It records the systemd unit's own
state, and fails the `the shipped supervisor unit is installed for an acceptance
run` check when `/etc/systemd/system/sbarbase.service` is not installed, so a
green acceptance run means the unit was present and enabled. The unit it verifies with `systemd-analyze` is
the installed file itself, and the evidence says which file that was; the
checkout's template is verified only when no unit is installed, and the evidence
records that too.

## Verify after install

- `lab/console_build_check.py` rebuilds the console and proves the served page is
  intact (non-empty index, every local asset it references present and hashed).
  The installer now refuses a build that produces an unusable page, and the
  rehearsal records `built console page is intact` before it starts anything.
- `lab/pinned_images_check.py` proves every pin is present locally and resolves
  to the pinned digest. It never pulls, and a tag that now points at a different
  digest fails instead of passing.

```
/usr/bin/python3 lab/install_server.py smoke         # management Auth, per-environment routes, console pid
bun lab/combined-gateway-check.ts                    # 14 simultaneous gateway checks (needs the console built)
/usr/bin/python3 lab/combined-supervisor-check.py   # full rehearsal: start, checks, supervised shutdown
/usr/bin/python3 lab/deployment_rehearsal.py         # one command: install, supervise, verify, shut down, evidence
/usr/bin/python3 lab/target_placement_rehearsal.py   # retained target placement: start, probes, stop
```

`deployment_rehearsal.py` writes `docs/evidence/deployment-rehearsal.json` and
exits non-zero on any failure; on a host without the required headroom it
records the preflight refusal and stops without starting anything.
`target_placement_rehearsal.py` exercises the adopted recovery-target placement
end to end and returns it to the paused, stopped state.

The smoke command reports each endpoint status and does not modify state. The
supervisor rehearsal additionally requires the same headroom as a normal start.

## HTTPS and network exposure

The console and the gateway bind loopback and are reachable only through the
host. Terminate TLS in a reverse proxy and keep the Docker networks internal: the
installer creates no published ports. Do not expose the management Auth endpoint
or the provisioning API directly. Set the public URL the console should advertise
in the proxy, not in the console build. Auth's email links and OAuth callbacks take
the public address from `SBARBASE_PUBLIC_URL` in a drop-in of the service
([sign-in settings](https://base.sbarah.com/en/docs/guides/sign-in/)).

Studio exists as an optional per-environment feature in the legacy shared-engine
workflow, with separately scoped CI evidence. Its proxy and authentication
requirements remain essential; native-dedicated Studio is not accepted by that
legacy result. Each environment's Studio has its own address. The shipped proxy
serves them with `--studio-host studio-{id}.example.com` (the same pattern as
`SBARBASE_STUDIO_HOST`), passing only names that match it with their own host, after a
wildcard DNS record and a certificate that covers it are in place
([Studio](https://base.sbarah.com/en/docs/guides/studio/#open-it-from-anywhere)); without it, Studio is reached through an
SSH tunnel. The proxy also reads `SBARBASE_STUDIO_HOST` when it has no
`--studio-host` flag, and `lab/studio_address.py apply DOMAIN` writes that setting
for the console and the proxy unit in one step. Studio has no login of its own and its pages carry a database connection
string, so it must never be reachable unauthenticated, and it must not ride the data
plane's route pattern. [Integration specification](https://github.com/M7MMAD-OMAR/sbarbase/blob/main/docs/engineering/STUDIO-INTEGRATION.md), sections 4
and 5, covers the gate and the routing options.

A reference termination ships with the repository and is exercised by the check
(`/usr/bin/python3 lab/tls_termination_check.py`, 27 checks,
`docs/evidence/tls-termination.json`). It needs only Bun and a certificate:

Pin the console's loopback port first, so the proxy's upstream survives restarts
and reboots. Without it the console picks a new ephemeral port at every start:

```
sudo mkdir -p /etc/systemd/system/sbarbase.service.d
printf '[Service]\nEnvironment=SBARBASE_CONSOLE_PORT=8787\n' | sudo tee /etc/systemd/system/sbarbase.service.d/console-port.conf
sudo systemctl daemon-reload && sudo systemctl restart sbarbase
```

```
bun deploy/console-tls-proxy.ts \
    --cert /etc/letsencrypt/live/console.example.com/fullchain.pem \
    --key  /etc/letsencrypt/live/console.example.com/privkey.pem \
    --public-host console.example.com \
    --upstream http://127.0.0.1:8787 \
    --https-port 8443 --http-port 8080
```

To keep it running across reboots, give it its own systemd unit that runs as the service
account. Name it `sbarbase-proxy.service`: `lab/studio_address.py apply` writes its
drop-in there unless `--proxy-service` names another unit. On Fedora, SELinux refuses a unit that executes Bun under `/home` directly
(status `203/EXEC`, permission denied), so start it through a shell:
`ExecStart=/bin/sh -c 'exec /home/sbarbase/.bun/bin/bun deploy/console-tls-proxy.ts ...'`.
The rehearsal VM ran it this way, with a local CA, across a reboot
(`docs/evidence/vm-https-first-project.json`). It redirects plain HTTP to
`https://<public host>/` on port 443, so publish it on 443 on a real server.

It refuses to start unless the certificate and key are regular files, the key is
not group or world readable, `--public-host` is set to a bare host name, and the
upstream is loopback. The loopback assertion is applied to the console URL
whichever way it was supplied, including the one read from
`.lab/upstream/server.json` when `--upstream` is omitted. It answers plain HTTP
with a 308 redirect to HTTPS, adds `Strict-Transport-Security`,
`X-Content-Type-Options`, `Referrer-Policy`, `X-Forwarded-Proto` and
`X-Forwarded-Host`, and logs only method, path and status: never bodies, query
strings, cookies or credentials. The forwarding headers carry `--public-host`,
never the client's own `X-Forwarded-*` or `Host` values, which are dropped
before the request reaches the console. It serves only `--public-host`, loopback
names and Studio addresses that match `--studio-host`, and answers any other host
with 421, so a wildcard DNS record never shows the console under another name.

Its own hardening is part of the checks: the redirect and the forwarded host come
from `--public-host`, never from the client's `Host` header (an attacker supplied
host cannot turn the redirect into an open redirect), hop by hop headers are
stripped before forwarding, and a request body over `--max-body` (the upload
limit plus 1 MiB by default) is answered `413` as soon as the stream passes the cap, without
buffering it in full. An operator may
prefer nginx, Caddy or the platform proxy; the checks above state which behaviour
any replacement must keep.

### Firewall: two routes from containers to the host

Two services in containers call the host on its Docker bridge addresses: Studio reaches
Auth and Storage through its route on port 25432, and Edge Functions reach Storage on
port 25433. A host firewall that drops incoming traffic by default (ufw's default) drops
these too, and Studio's Storage, users and API pages then fail. Allow exactly these two,
which keeps SSH, 80 and 443 the only ports open from outside:

```
SUBNET=$(docker network inspect sbarbase-durable-net --format '{{(index .IPAM.Config 0).Subnet}}')
GATEWAY=$(docker network inspect sbarbase-durable-net --format '{{(index .IPAM.Config 0).Gateway}}')
sudo ufw allow from "$SUBNET" to "$GATEWAY" port 25432 proto tcp comment 'sbarbase Studio route'
sudo ufw allow in on br-+ to any port 25433 proto tcp comment 'sbarbase Functions to Storage'
```

The first rule admits only the runtime network, so a container on another network cannot
reach the Studio route. The second admits any Docker bridge on that one port; each
listener then refuses sources outside its own environment's network. Opening Studio
checks the first route from inside Studio and stops with a message naming the firewall
when it is blocked.

## If a restore is interrupted

A failed restore leaves `.lab/upstream/recovery-target.json`, and a plain rerun
refuses because the descriptor exists. Do not edit that state by hand:

```
/usr/bin/python3 lab/recovery_reconcile.py        # stop the retained containers
/usr/bin/python3 lab/retire_recovery_target.py --reason "why it failed"
/usr/bin/python3 lab/recovery-restore-db.py       # now a fresh restore may run
```

Retirement refuses unless the descriptor's status is `interrupted`, `failed` or
`cleanup-failed`, every container of that prefix is stopped, and no per-target
HBA operation is pending. It archives the descriptor into
`recovery-target-history/`, records the path in the cutover journal, and removes
the active descriptor; containers and volumes are never deleted. A target that
already holds a usable database can instead be adopted in place with
`lab/adopt-retained.py target`.

## Backup, upgrade and rollback

The step by step versions are [backup and restore](https://base.sbarah.com/en/docs/guides/backup-and-restore/) and [upgrades](https://base.sbarah.com/en/docs/guides/upgrades/).

- Backup: every environment, Storage's shared metadata and the management
  database are backed up daily while Sbarbase runs (`lab/backup.py`,
  `SBARBASE_BACKUP_HOUR`), and `lab/offsite.py` sends encrypted copies to an
  S3-compatible bucket. The encrypted export and independent-restore path is
  documented in [INDEPENDENT-RESTORE](https://github.com/M7MMAD-OMAR/sbarbase/blob/main/docs/engineering/INDEPENDENT-RESTORE.md), a
  historical independent-restore fixture. Separate legacy Docker and local-VM
  backup/restore observations are recorded in [status](https://base.sbarah.com/en/docs/reference/status/);
  none establishes complete candidate or physical fresh-host recovery.
- Upgrade: the console's Updates page shows a newer signed release and installs a
  safe one with one click; `lab/upgrade.py` does the same from the command line.
  Each upgrade backs up every environment first, holds application traffic until
  the new version passes its health checks, and moves back by itself if it does
  not. Automatic updates are opt-in. The [upgrades guide](https://base.sbarah.com/en/docs/guides/upgrades/) has the
  steps, the classes that need a rebuild or a migration, and the limits.
- Rollback: before confirmation it is automatic. After it, **Roll back** on the
  Updates page or `lab/upgrade.py rollback` keeps everything written since the
  update, and refuses when the previous version cannot open the control catalog.
- Changing a pinned upstream image is a maintainer's task: follow
  [UPSTREAM-UPDATE-POLICY](https://github.com/M7MMAD-OMAR/sbarbase/blob/main/docs/engineering/UPSTREAM-UPDATE-POLICY.md), which records
  the rollback pin and any data migration in a dated entry under `docs/upstream/`.

## Historical rehearsal evidence and current limits

- No end-to-end install rehearsal on an independent public server is established. An empty
  server has been simulated in a local virtual machine (Fedora 44 Cloud, clean
  clone, `lab/vm-rehearsal.sh`), whose later rehearsal found ten defects the
  development host could not show; see
  [deployment readiness](https://base.sbarah.com/en/docs/reference/deployment-readiness/). The
  documented command itself completed end to end on the development host at
  7:25 PM, 13 of 13, including the unit install as root and its restart
  (`docs/evidence/server-acceptance-latest.json`); what a server adds is root,
  the service account, a public certificate and the absence of a retained
  installation. The preflight, the runtime startup, adoption and verification are
  also each proven separately in the evidence files under `docs/evidence/`.
- No production capacity claim: 5888 MiB and 5.75 CPUs are configured ceilings,
  not measured peak demand. Sustained mixed load and 10/100-project capacity are
  unproven.
- Realtime, Functions and Studio have optional implementations in the legacy
  workflow, with separately scoped CI observations. They are not thereby
  accepted for native-dedicated placement. The connection pooler and exposed
  operator Cron feature remain unimplemented in that legacy workflow; accepted
  original-image Cron/HTTP fixtures are a separate bounded prerequisite.
- Legacy encrypted copy/fetch/restore and second-VM restore have scoped
  historical observations. Complete fresh-host installation recovery, candidate
  physical restore and coordinated multi-host operation remain unaccepted.
- Updates from the console and opt-in automatic updates exist, with a health-gated
  way back and a start guard ([upgrades](https://base.sbarah.com/en/docs/guides/upgrades/)). They are covered by unit
  tests plus historical clean-runner CI and local-VM observations summarized in
  [status](https://base.sbarah.com/en/docs/reference/status/#update-channel-2026-09-25). On the real server,
  signed releases from the canonical repository with the maintainer's key were
  installed from the command line (`lab/upgrade.py start --release`, first 0.3.1);
  the console's install button, automatic updates and **Roll back** have not run
  there.
- After an unclean supervisor stop, `lab/leftover_runtime.py` provides a
  lock- and receipt-gated stop of owned containers before preflight, retaining
  volumes. This correction has source regression evidence only; no live kill
  rehearsal is established. A live owner or pending receipt/journal still
  refuses reconciliation. Reinstall the unit to gain its documented preflight
  cleanup step; this does not establish arbitrary crash recovery.
- Bootstrap creates the first operator; invitations are a separate legacy
  feature with scoped local-VM evidence. The console has sign-in factors (MFA)
  and rate limits on its sign-in, which answer `429` when exceeded. See
  [status](https://base.sbarah.com/en/docs/reference/status/) for the historical scope.

## Private directory permissions

Installation and startup create `.secrets` and `.secrets/upstream` with mode `0700`, owned by the account running the service. Symlinks and existing directories with a different owner or mode are refused. The runtime state directory `.lab/upstream` is also kept `0700`: unlike the secrets, one left open by an install under umask 002 is closed at start, because restore refuses its journal in an open directory; a link or another owner is refused. If a new version refuses an older checkout whose parent was created as `0755`, stop the installation normally and verify both directories are real directories owned by the service account before setting only these two directory modes to `0700`. The shipped Docker image runs as root inside the container. This repair does not require changing ownership or internal file modes; retain all secret data and backups. `Runtime private directory ownership or mode refused` and `Runtime private directory identity changed` require correcting permissions or the checkout path before retrying.