Deployment behavior-to-test contract (#4305)
Written before implementation. Risk is high: infrastructure lifecycle, database mutation/recovery, RBAC/secret boundaries, and a published deployment interface. The contract is maintenance-only v1.
Every row has the ten required design fields. Commands name executable test
entrypoints; the integration ledger below distinguishes completed evidence from
required qualification. A PENDING: downstream rollout needs separate target
authorization and is not claimed complete. Rendering proves manifest behavior only. Mock observations
cannot replace the disposable controller/database exercise or real rollout.
In the table, PG means the initial deployed PostgreSQL boundary (14+ under the
existing permission contract). The controller feature floor is helm-controller
1.4.5 with matching CRDs; record actual tested versions for each qualification. SQLite/DuckDB/PGlite are not deployed chart targets. A public
core API supporting those dialects needs its own package matrix across every
declared supported dialect before shipping.
| Behavior / invariant | Reachable trigger | Positive case | Negative / failure case | Actor / context | Data executor / transaction | Supported runtime / dialect | External contract edge | Test level | Validation command |
|---|---|---|---|---|---|---|---|---|---|
| Existing database/bucket retained on install, failure and uninstall | Bootstrap release against sentinel resources, then uninstall | Same provider identity and sentinel contents remain | Ownership conflict or attempted provisioning refuses without deletion | App releaser; platform-owned providers | PG sentinel transaction; bucket object roundtrip; chart owns neither | Node/Kubernetes; PG | Existing Secret/provider bindings and Helm ownership | Render + disposable integration | node --test scripts/deployment-chart.test.mjs; SMRT_DEPLOYMENT_EXERCISE=1 KUBECONFIG=<isolated-kind> node scripts/deployment-exercise.mjs |
| Explicit adoption preserves unrelated objects | Adopt existing names into chart | Reviewed ownership transfer only; independent apps remain in their own namespace | Unreviewed name collision refuses; unrelated Studio object unchanged | Releaser and IaC owner; dedicated application namespace | N/A: metadata ownership, no app SQL | Helm/Kubernetes | Existing labels, annotations, field owners | Contract + integration | node --test scripts/deployment-release.test.mjs; SMRT_DEPLOYMENT_EXERCISE=1 KUBECONFIG=<isolated-kind> node scripts/deployment-exercise.mjs |
| Strict validated values and complete preflight before maintenance | Render/plan candidate values | Valid optional omission, multiple digest images and complete inventory pass | Unknown enum/field, malformed input, mutable image, secret literal, unsupported capability, missing Secret/key/RBAC/admission fails before route change | Authorized releaser with allowed and denied service accounts | N/A: validation and read-only resource inventory | Node/Helm/Kubernetes | Schema, RBAC, admission, CRDs, pull/config/secret refs; upstream failure | Unit + render + integration | node --test scripts/deployment-chart.test.mjs scripts/deployment-release.test.mjs |
| Named independent workloads and singleton scheduling | Render Iolaus and Work fixtures | Distinct images/commands, capacity, services, probes, node constraints, scratch/drain retained; combined worker singleton | Schedule replicas >1 and missing required readiness refuse | Application declaring web, combined scheduler, workers, monitor | N/A: manifest generation | Helm/Kubernetes | Workload role enum and field omissions | Render contract | helm template iolaus charts/smrt-app -f charts/smrt-app/fixtures/iolaus.json; helm template work charts/smrt-app -f charts/smrt-app/fixtures/work.json |
| Maintenance independent of application | Bootstrap or upgrade maintenance phase | Public hostname returns expected 503 while app stops | Wrong host/body/backend or responder unavailable blocks advancement | External client, ingress/controller | N/A: static HTTP, no DB | Kubernetes/HTTP | TLS/DNS/ingress and independent responder | Integration | SMRT_DEPLOYMENT_EXERCISE=1 KUBECONFIG=<isolated-kind> node scripts/deployment-exercise.mjs |
| Old images preserved until quiesced | Publish target with maintenance desired | Running current image unchanged until actual stop | Early target rollout or revision skipping rejected | Releaser, Flux, existing runtime | N/A: runtime identity observation | Kubernetes/Flux | Helm rendered image, pod imageID and accepted Git revision | State-machine + integration | node --test scripts/deployment-release.test.mjs; SMRT_DEPLOYMENT_EXERCISE=1 KUBECONFIG=<isolated-kind> node scripts/deployment-exercise.mjs |
| Complete writers stopped/fenced; monitors classified accurately | Stop phase with Deployments, CronJobs and active Jobs | Termination/drain and platform fencing proven for complete inventory; restricted monitor identified | Desired zero with terminating pod, unknown/external writer, active Job, timeout or unobservable inventory blocks migration | Runtime/migration/monitor identities, namespace and external clients | PG runtime sessions and migration lock; read-only monitor privileges | Kubernetes/PG | Writer inventory, SIGTERM grace, CronJob suspend, RBAC and fence | Contract + disposable integration | node --test scripts/deployment-release.test.mjs; SMRT_DEPLOYMENT_EXERCISE=1 KUBECONFIG=<isolated-kind> node scripts/deployment-exercise.mjs |
| Migrator lock and static/runtime guard precede writes | Migration Job starts/restarts | Exclusive migrator and agreed inventory allow planned mutations | Lock contention, new writer, mismatch, missing permissions or wrong image causes zero mutation | Migration-only role; runtime excluded | PG lock plus supported SQL MigrationTracker transaction boundaries | Node/PG | Dedicated Job image/command, registry and schema inventory | Package + integration | SMRT_TEST_POSTGRES_URL=<isolated> pnpm --filter @happyvertical/smrt-core exec vitest run src/migrations/__tests__/deployment-phase-tracker-postgres.optional.test.ts src/migrations/__tests__/deployment-roles-postgres.optional.test.ts; SMRT_DEPLOYMENT_EXERCISE=1 KUBECONFIG=<isolated-kind> node scripts/deployment-exercise.mjs |
| Owning migration instructions assign schema creation to migration, before runtime | Follow the historical Products SKU upgrade instructions | Reviewed migration identity creates required schema before startup; runtime checks existence | A runtime boot is presented as an application-schema creator | Application operator and documentation reader | Reviewed migration executor; runtime table verifier performs no DDL | Node/application schema; historical copy/drop SQL is not requalified here | Products guide and core table-verifier contract | Documentation/source contract | pnpm knowledge:check --strict --format markdown; pnpm check:agents-chain; pnpm check:standards |
| Transactional data step marker is atomic | Interrupt before marker, after marker, before/after commit | Data and checksum marker commit together; completed step skipped on resume | Failure rolls both back; wrong executor affinity and checksum mismatch refuse | Serialized migration owner | Same PG transaction handle for mutation and marker; never outer db | Node/PG; owning API dialects must be enumerated | Tracker persistence and connection failures | Package persistence + disposable integration | SMRT_TEST_POSTGRES_URL=<isolated> pnpm --filter @happyvertical/smrt-core exec vitest run src/migrations/__tests__/deployment-phase-tracker-postgres.optional.test.ts src/migrations/__tests__/deployment-roles-postgres.optional.test.ts |
| Committed steps survive later migration failure | Commit A, fail B, restart retained Job | A preserved and skipped; B resumes; seed/verify ordered after dependencies | Repeated execution, changed A checksum, partial B failure, missing marker blocked or reconciled | Migration owner after process death | SQL MigrationTracker plus durable data ledger; explicit transaction per step | Node/PostgreSQL only | Process interruption and persistent database state | Persistence + integration | SMRT_TEST_POSTGRES_URL=<isolated> pnpm --filter @happyvertical/smrt-core exec vitest run src/migrations/__tests__/deployment-phase-tracker-postgres.optional.test.ts src/migrations/__tests__/deployment-roles-postgres.optional.test.ts; SMRT_DEPLOYMENT_EXERCISE=1 KUBECONFIG=<isolated-kind> node scripts/deployment-exercise.mjs |
| Nontransactional ambiguity never silently reruns | Kill external/nontransactional phase between side effect and marker | Idempotent checkpoint reconciliation or explicit needs-repair | Ambiguous partial side effect cannot authorize completion or blind retry | Migration owner/operator; external service | N/A transaction: nontransactional by definition; durable checkpoint required | Node/PG plus declared external target | Upstream failure, network loss, repeated execution | Contract + fault injection | SMRT_TEST_POSTGRES_URL=<isolated> pnpm --filter @happyvertical/smrt-core exec vitest run src/migrations/__tests__/deployment-phase-tracker-postgres.optional.test.ts src/migrations/__tests__/deployment-roles-postgres.optional.test.ts |
| Resume from accepted state at every phase | Kill runner in each phase and restart on another process | Git + live evidence + durable ledger determine next safe action | Missing/stale evidence, local-only success, superseded release, stale Ready and skipped Git revision cannot advance | Authorized runner in same app/environment | PG ledger read; Git CAS for desired state | Node/Git/Flux/Kubernetes | Revision acceptance, generation and evidence identity | State-machine + integration | node --test scripts/deployment-release.test.mjs |
| Concurrent releases serialize and preserve unrelated Git updates | Two runners target same app; third changes another app | One release owner; CAS rebases unrelated changes safely | Competing phase/release, forced stale update or lost unrelated change refuses | Two authorized releasers and unrelated monorepo actor | Git ref CAS; database migration lock is separate | Node/Git | Protected ref, remote fetch/push failure | Local-Git integration + state-machine | node --test scripts/deployment-release.test.mjs |
| Flux drift and Helm failure cannot reactivate old writers/traffic | Reconcile repeatedly while stopped or migration fails | Maintenance/stopped desired state persists; retained failure evidence | Imperative drift repair or automatic rollback restores active old release: fail | Flux/Helm controllers; release runner | PG sentinel unchanged after failed step; no old writes | Qualified Flux/Helm/Kubernetes; PG | Controller remediation APIs/version and protected writes | Disposable controller integration | SMRT_DEPLOYMENT_EXERCISE=1 KUBECONFIG=<isolated-kind> node scripts/deployment-exercise.mjs |
| Verification binds exact images and actual behavior | Start target behind maintenance | All required generations/imageIDs/endpoints and workers ready; app assertions pass | Old healthy endpoint, stale Ready, self-reported digest only, wrong image, missing/duplicate workload, alive HTTP500 fail | Internal verifier; new runtime/worker identities | Runtime PG role, read-only schema readiness; no migration credentials | Kubernetes/HTTP/PG | EndpointSlice/pod identity, readiness, worker heartbeat/drain | Contract + integration | node --test scripts/deployment-release.test.mjs; SMRT_DEPLOYMENT_EXERCISE=1 KUBECONFIG=<isolated-kind> node scripts/deployment-exercise.mjs |
| Activation/failure recovery respects changed DB | Switch public route; fail serving assertion | Target serving matches verified evidence; failure returns maintenance | Automatic old-image restore or corrective migration with new writers active refuses | Public verifier and authorized recovery operator | PG runtime writes; stop/fence before corrective mutation | Flux/Kubernetes/HTTP/PG | Routing revision, public serving target, repair transition | State-machine + disposable integration | node --test scripts/deployment-release.test.mjs; SMRT_DEPLOYMENT_EXERCISE=1 KUBECONFIG=<isolated-kind> node scripts/deployment-exercise.mjs |
| Secrets and tenant/role boundaries remain private | Plan, fail, status and execute with synthetic credential canaries | Only references/redacted diagnostics; migration/runtime/monitor allow/deny proofs; worker-only secret absent from web | Credential URL/error leakage, migration secret in runtime, monitor write or undeclared read fails | Migration/runtime/monitor roles; allowed/denied resource and active application context | PG role-specific grants; permission apply atomic verification | Node/Kubernetes/PG | Secret refs, logs, provider failure text, SQL ACL and RBAC | Unit + permission integration | node --test scripts/deployment-release.test.mjs; SMRT_TEST_POSTGRES_URL=<isolated> pnpm --filter @happyvertical/smrt-core exec vitest run src/migrations/__tests__/deployment-phase-tracker-postgres.optional.test.ts src/migrations/__tests__/deployment-roles-postgres.optional.test.ts; existing permission package tests remain prerequisites |
| Artifact validation remains identity-bound | Package chart and associate source/config/image/validation | Versioned chart installs; evidence matches tree/lock/toolchain/images | Missing/incorrect #4306 receipt never bypasses checks; mutable tag association refuses | Publisher/releaser | N/A: build metadata, no app SQL | Helm/Node/OCI | Registry publication, source and validation identity | Packaging + contract | helm package charts/smrt-app; node --test scripts/deployment-release.test.mjs |
| Real downstream adoption is distinguished from fixture success | Select and authorize a concrete application/environment | Reviewed downstream changes deploy; actual accepted phases/new image and sentinel/recovery evidence recorded | Fixture render or mock run labeled rollout; unauthorized live mutation | App owner, platform owner, authorized releaser | Target PG/bucket under approved maintenance/recovery plan | Exact deployed tool/controller/database versions | Separate repositories, protected deployment and operational authorization | Authorized real application rollout | PENDING: target-specific reviewed rollout command and evidence |
This documentation alignment covers migration-before-runtime responsibilities; it does not requalify the historical Products copy/drop SQL.
Qualification evidence
This matrix was drafted before implementation. Record current results alongside an exact reviewed revision in the PR and retained validation artifacts, including:
- Helm schema/render/packaging and runner contracts, including local Git races.
- Real PostgreSQL durable-step and role-boundary tests, including interruption.
- The isolated Flux/Kubernetes/Ingress/PostgreSQL/S3 exercise, its tool/controller versions, sentinel retention, failure/resume and cleanup outcomes.
- The required touched-package/root validation and independent high-risk review.
The exercise uses synthetic applications and test-only HelmRelease changes. It does not prove protected Git publication in a real cluster, downstream wrapper adoption, production writer fencing, or a real downstream rollout. Those require reviewed downstream changes and explicit target/environment authorization. Keep #4305 open until its actual adoption and rollout criteria have evidence.
Run focused package checks first and include focused deployment commands in normal
validation. Full validation follows root AGENTS.md and TESTING_STANDARD.md;
heavy work uses the prescribed cluster-test workflow. The integration owner must
record disposable teardown even on failure. Do not replace failed documented
commands with a different toolchain and report green.
| Chart 0.2.0 boundary | Positive / negative | Actor and evidence |
|---|---|---|
| Security and observer schema/render | Bounded non-root contexts, scratch and explicit observer SA render; root/privilege/unknown fields, runtime observer or missing SA fail | Helm; node --test scripts/deployment-chart.test.mjs |
| Platform preflight | Declared SA/CA/read-only RBAC and synthetic Pod admission pass; missing SA/CA/RBAC/create permission/admission fail before maintenance | Runner, reconciler and observer; node --test scripts/deployment-release.test.mjs |
| Actual Pod envelope | Exact account/security/volumes pass; changed IDs, account, added token volumes or host/privilege escape fail | Actual Pod identity; same runner suite |
| Hardened execution | Web/task/migration execute with non-root 10001, RuntimeDefault, dropped ALL, no escalation, read-only root and /tmp emptyDir | Existing disposable deployment exercise; integration owner runs isolated stack |