# Provisioning

## What it is

Provisioning is the background work that turns "create environment" into a running database, logins, connection rules and services. It is written so that a crash at any moment never causes the same work to run twice on a guess; when the outcome is unknown, it stops and asks for an operator.

## Why

**Choice: record before acting, refuse when unsure.** Creating an environment touches Docker, PostgreSQL and files. If the worker dies halfway, simply retrying could create a second database, reuse a half-made login, or overwrite connection rules another operation just wrote. So every external effect is preceded by a durable record, and after a crash the worker settles only what it can prove.

**Rejected alternative: idempotent retry.** "Just run it again, it is idempotent" assumes every step can detect its own partial result. Several cannot (a database created but not yet closed to other logins, a reload that was sent but not confirmed), and guessing wrong on a shared engine can hurt a neighbour.

## How we built it

![Five steps: queue a job, one worker claims it, it writes an effect receipt to disk, runs the fenced effect, records the outcome and consumes the receipt; after a crash a completed receipt is settled, an interruption proven to precede any change is requeued, and an unknown outcome stops for an operator](https://base.sbarah.com/docs/diagrams/provisioning.svg)

*Record first, act, then settle. An unknown outcome never runs twice on a guess.*

The same exchange as a sequence:

![A sequence between the control API, the catalog, the worker with its exclusive lock, the receipt file and Docker with SQL: queue a job, claim it, write the receipt and flush it, run the fenced effect, get a known outcome, record it with the job completion, consume the receipt; after a crash a completed receipt is settled and an unknown one stops](https://base.sbarah.com/docs/diagrams/provisioning-sequence.svg)

*The receipt is on disk before the effect runs, and consumed only after the outcome is in the catalog.*

The pieces, in plain words (each term is also in the [glossary](https://base.sbarah.com/en/docs/reference/glossary/)):

- **Worker lock.** Only one provisioning worker runs at a time; it holds an exclusive file lock.
- **Effect receipt.** Before an effect starts, the worker writes a small file naming the exact job claim and attempt, and flushes it to disk. It never overwrites an existing one. A completed receipt is settled against the catalog on restart; a pending one with no proof of outcome blocks startup and any replay.
- **Guardian and witness.** The effect runs under a guardian process with a deadline. The native provisioner writes a completion witness when it finishes, so a lost acknowledgment can be recovered without re-running the work.
- **Fencing.** SQL and connection-rule changes carry an operation token. Once a token is revoked (a tombstone is written), a stale process holding it cannot write again, even if it wakes up later.
- **Generation pin.** Each managed database container is pinned by its exact identity. Startup refuses a container that changed underneath it, and only `lab/migrate-generation.py` may replace one, through its own journaled and crash-tested steps.
- **Preflight recovery.** An interruption proven to have happened before any external change can be requeued automatically, a bounded number of times.

Code: [lab/worker.py](https://github.com/M7MMAD-OMAR/sbarbase/blob/main/lab/worker.py), [lab/effect_receipt.py](https://github.com/M7MMAD-OMAR/sbarbase/blob/main/lab/effect_receipt.py), [lab/effect_lease.py](https://github.com/M7MMAD-OMAR/sbarbase/blob/main/lab/effect_lease.py), [lab/provision.py](https://github.com/M7MMAD-OMAR/sbarbase/blob/main/lab/provision.py), [lab/sql_operation_fence.py](https://github.com/M7MMAD-OMAR/sbarbase/blob/main/lab/sql_operation_fence.py), [lab/hba_authority.py](https://github.com/M7MMAD-OMAR/sbarbase/blob/main/lab/hba_authority.py), [lab/hba_generation.py](https://github.com/M7MMAD-OMAR/sbarbase/blob/main/lab/hba_generation.py), [lab/migrate-generation.py](https://github.com/M7MMAD-OMAR/sbarbase/blob/main/lab/migrate-generation.py). The read-only inspector is [lab/inspect-provisioning.py](https://github.com/M7MMAD-OMAR/sbarbase/blob/main/lab/inspect-provisioning.py).

## Limits

- This is replay prevention, not complete automatic recovery. A later-stage unknown outcome stays blocked until an operator reconciles it; the bounded reconciliation workflow for those cases is not built.
- Crash tests cover named checkpoints on disposable fixtures, not arbitrary crashes, Docker daemon failure or power loss.
- A killed provisioner may leave a Docker client or daemon-side effect running; the pending receipt only stops the worker from repeating it.
- Trusted operator code and direct Docker or SQL access are outside this protection.
- Never delete a pending receipt, journal, generation pin or tombstone to get past a refusal.

## Go deeper

- [Provisioning receipts](https://github.com/M7MMAD-OMAR/sbarbase/blob/main/docs/engineering/PROVISIONING-RECEIPTS.md), [effect guardian](https://github.com/M7MMAD-OMAR/sbarbase/blob/main/docs/engineering/EFFECT-GUARDIAN.md), [native outcome recovery](https://github.com/M7MMAD-OMAR/sbarbase/blob/main/docs/engineering/NATIVE-OUTCOME-RECOVERY.md), [preflight recovery](https://github.com/M7MMAD-OMAR/sbarbase/blob/main/docs/engineering/PREFLIGHT-RECOVERY.md).
- [SQL operation fence](https://github.com/M7MMAD-OMAR/sbarbase/blob/main/docs/engineering/SQL-OPERATION-FENCE.md), [HBA operation authority](https://github.com/M7MMAD-OMAR/sbarbase/blob/main/docs/engineering/HBA-OPERATION-AUTHORITY-DESIGN.md), [source HBA integration](https://github.com/M7MMAD-OMAR/sbarbase/blob/main/docs/engineering/SOURCE-HBA-INTEGRATION.md).
- [Container generation migration](https://github.com/M7MMAD-OMAR/sbarbase/blob/main/docs/engineering/CONTAINER-GENERATION-MIGRATION.md) and the [provisioning mutation map](https://github.com/M7MMAD-OMAR/sbarbase/blob/main/docs/engineering/PROVISIONING-MUTATION-MAP.md).