CI/CD topology
Four environments: Local (SQLite on the host, DynamoDB Local in Docker) → Development (Docker + DynamoDB Local) → Test (SAM) → Production (SAM, manual).
Catalog: docs/README.md. Env vars: docs/web/setup/environment.md. Docker: docker.md. SAM: infra.md.
Agent skills (this repo):
- Operate / debug AWS SAM CD + Docker parity + skip-ci recovery:
.agents/skills/portfolio-aws-cicd/
Portable templates for other projects / AI agents:
- CI/CD: templates/nuxt-ssr-cicd/ ·
.agents/skills/nuxt-ssr-cicd-template/ - Commit / PR / rulesets: templates/commit-pr-rules/ ·
.agents/skills/commit-pr-rules-template/
Flow
PR → development → CI: Local parity only (skip version-bump-only PRs)
merge → development → Regression: Full regression (e2e) ← the only e2e
CD AWS Test (dry-run): one job, SYS_ENV=test ← the only test dry-run
Version bump (development): patch PR (skipped while a release PR is open)
npm run release:pr → release/v<version>: ONE commit = development's tree on top of main
PR release/v* → main → CI: Local parity + Regression gate (regression-gate.yml; reads the post-merge e2e; runs no tests)
Version bump (release): minor|major PR into the release branch, lands before the merge
squash-merge → main → CD AWS Test (SAM deploy → test); on success → production dry-run
(no full CI re-run; prod dry-run does not deploy)
Version sync: bump PR into development carrying main's release version
release published → CD AWS Production (tagged commit + Environment approval)
workflow_dispatch → CD AWS Production (confirm=deploy + Environment approval)
Pull requests never run e2e or a SAM dry-run. Each dry-run runs once per target: test after a merge to development, production after a successful test deploy on main. A legacy release PR straight from development has the development tip as its head, so GitHub also lists that commit’s push checks on it, labelled (push); a release PR from npm run release:pr does not (its head is a new commit). scripts/cicd-workflow-policy.test.mjs fails if any pull_request-triggered workflow runs e2e:regression or sam-dry-run.
Merge policy
| Branch | How to land changes | Merge method |
|---|---|---|
development |
Pull request only (admin bypass allowed) | Merge or squash |
main |
Pull request only (admin bypass allowed) | Squash only (workflows assume it) |
Configure via Rulesets (npm run gh:setup-rulesets). Squash commit text must be PR title only (npm run gh:setup-merge-settings) — see Skip-CI trap below. Do not treat unprotected direct push as the default path.
Matrix
| Environment | Runtime | Data | Trigger | GitHub Environment |
|---|---|---|---|---|
| Local | Nitro node-server |
SQLite data/local.sqlite |
Developer machine | — |
| Development | Docker web + api + DynamoDB Local + MinIO (local Compose; Postgres opt-in) |
DynamoDB Local (default), PostgreSQL via DATA_BACKEND=postgres |
Local npm run docker:dev only |
— |
| Test | SAM: Lambda + HTTP API + S3 + DynamoDB | DynamoDB | Squash-merge to main (after PR CI) |
test |
| Production | Same SAM topology | DynamoDB (prod stack) | GitHub Release published (non-prerelease) or manual workflow_dispatch |
production |
Docker Compose under docker/ maps to this matrix as local stand-ins: local ≈ docker-compose.local.yml (web + DynamoDB Local + MinIO); development ≈ docker-compose.dev.yml (web/api + DynamoDB Local + MinIO; Postgres opt-in); test ≈ docker-compose.test.yml (DynamoDB Local + MinIO + Lambda RIE). All four stacks share docker/compose.aws-emulators.yml. None is identical to AWS (no CloudFront in Compose; split web/api only in Docker dev). DATA_BACKEND overrides the SYS_ENV data-store default. docker:local and docker:dev both bind host 4200 — run only one. Crosswalk table and # aws: service comments: docker/README.md.
Nitro output (SYS_ENV)
Every Nuxt build writes to workspace .output/<sysEnv> (scripts/nitro-output-dir.mjs). Do not mix trees.
SYS_ENV |
Preset | Directory | Consumers |
|---|---|---|---|
local |
node-server |
.output/local |
npm run build / CI Local parity |
development |
node-server |
.output/development |
Docker Dockerfile.app (COPY flattened to /app/.output) |
test |
aws_lambda (e2e: node-server) |
.output/test |
sam:build test, docker:test, CI e2e |
production |
aws_lambda |
.output/production |
sam:build production / CD prod |
E2E keeps SYS_ENV=test identity and still lands in .output/test/ when NITRO_PRESET=node-server. The message and blog API store mappers give E2E_STORE_SYS_ENV=local precedence over the public test identity so the no-Docker regression job uses SQLite; scripts/e2e-regression.mjs additionally pins DATA_BACKEND=sqlite because SYS_ENV=test would otherwise select DynamoDB. It also drops NUXT_APP_CDN_URL / NUXT_PUBLIC_CDN_URL when they come from a local .env.test: that CDN doesn’t hold the fresh build’s JS chunks, so the app would never hydrate. Set them explicitly to keep them. @tgmc/web build inputs include SYS_ENV and NITRO_PRESET so Nx caches do not collide.
Workflows
| Workflow | Purpose |
|---|---|
.github/workflows/ci.yml |
PR-only local parity (Nx affected + Nx Cloud, Node 24, SYS_ENV=local → .output/local). Skips version-bump-only PRs. On PRs to development, when web/docker are affected, also runs SYS_ENV=development npx nx run @tgmc/web:build (.output/development). No e2e, no dry-run. workflow_dispatch (no inputs) exists for the version-bump workflows: on a ci/version-bump-* branch it short-circuits green; on any other ref (a release head) it diffs against main. Exposes deployable via workflow_call. |
.github/workflows/regression-gate.yml |
PRs to main (and dispatch on a release head) only: job Regression gate (scripts/regression-gate.js, see Regression gate) — runs no tests, fails when the release version bump has not landed, then waits for the post-merge Full regression (e2e) run on development and passes only when it succeeded (≤ 45 min wait). Its own workflow, so PRs into development list no skipped gate check. |
.github/workflows/regression.yml |
On push to development (and manual dispatch) only: Full regression (e2e) via npm run e2e:regression with SYS_ENV=test identity on node-server (serves .output/test; no Docker for the app in GHA; SQLite via E2E_STORE_SYS_ENV=local and DATA_BACKEND=sqlite, which scripts/e2e-regression.mjs sets); timeout-minutes: 35. Version-bump merges start no push runs (GITHUB_TOKEN). Never runs in PRs. |
.github/workflows/cd-aws-test-dry-run.yml |
On push to development only (nowhere else): one job — deployable gate, then AWS Test SAM dry-run (sam-dry-run.mjs test) when deployable. No full CI, no deploy, no GitHub Environment. Replaces the old cd-development.yml name. |
.github/workflows/version-bump.yml |
Version bump (development) — push → development: patch-only bump PR, landed by .github/actions/land-version-bump. Skipped while a release PR (development → main) is open, so a release gets one bump. |
.github/workflows/version-bump-release.yml |
Version bump (release) — PR → main: minor|major bump PR into the release PR head (the release/v* branch), landed by .github/actions/land-version-bump before the main merge. Never downgrades; no-op when the head already carries the release version. |
.github/workflows/version-sync.yml |
Version sync — push → main: when main’s version is above development’s (a release landed its bump on the release branch), opens chore: bump version to <main version> (release sync) into development through the same land action. No-op otherwise; never downgrades. |
.github/workflows/cd-aws.yml |
On main (or manual): deployable gate → SAM deploy to test + S3 sync + smoke → on success production SAM dry-run (sam-dry-run.mjs production, no deploy). Does not re-run full CI. Manual workflow_dispatch always deploys test then dry-runs prod. |
.github/workflows/cd-aws-prod.yml |
On Release published (non-prerelease): SAM deploy production from the release tag + Environment reviewers. Manual workflow_dispatch still requires confirm=deploy. |
.github/workflows/sync-tamaramack-github-io.yml |
Exports the tracked main source commit and opens or updates a destination PR from automation/portfolio-sync into development, replacing only apps/portfolio/. Automatic pushes require TAMARAMACK_GITHUB_IO_SYNC_ENABLED=1; manual dispatch remains available. |
.github/workflows/update-agents.yml |
Path-filtered AGENTS.md regen on development/main (Node 24, contents: write) |
.github/workflows/pages.yml |
Publishes docs/ to GitHub Pages (Jekyll Cayman) on push to development/main. The github-pages environment must allow both branches (npm run gh:setup-environments); otherwise the deploy job fails immediately with empty logs. |
Portfolio synchronization to tamaramack.github.io
.github/workflows/sync-tamaramack-github-io.yml publishes this repository’s tracked portfolio snapshot to tamaramack/tamaramack.github.io through a destination pull request. It uses git archive at the triggering source commit, records its UTC timestamp for deterministic destination commits, recreates automation/portfolio-sync from destination development, and opens or updates the matching pull request. Run-specific URLs appear in the pull-request body, not the snapshot provenance, so same-commit retries remain unchanged. It never deploys the destination directly.
Ownership boundary:
| Owner | Path or responsibility |
|---|---|
| Source repo | Exports all tracked content at the triggering commit. Untracked files and working-tree state are excluded. |
| Source repo | Replaces the complete destination apps/portfolio/ snapshot; that directory is disposable sync output. |
| Destination | Owns apps/profile/, the GitHub Pages root. The source workflow and sync helper must never change it. |
| Destination | Owns repository validation, promotion, and deployment workflows after the generated pull request is opened. |
The sync additionally fails closed unless the destination is the expected repository and branch, apps/profile/ exists, protected ancestors are not symbolic links, and every path in the entire staged diff is beneath apps/portfolio/. After opening or updating the PR, it retries for up to 45 seconds to confirm destination PR checks were created; it does not wait for them to pass.
Source repository setup:
- Create a fine-grained personal access token limited to the destination repository
tamaramack/tamaramack.github.ioonly. Grant Contents: Read and write, Pull requests: Read and write, and Metadata: Read; grant no broader repository access. - Store it in this source repository as the Actions secret
TAMARAMACK_GITHUB_IO_SYNC_TOKEN. - Use the source repository Actions variable
TAMARAMACK_GITHUB_IO_SYNC_ENABLEDas the automatic-run switch. Only the exact value1enables runs caused by pushes to sourcemain;workflow_dispatchis allowed regardless of the variable and still requires the token.
Rollout:
- Merge the workflow while automatic push runs remain gated by leaving
TAMARAMACK_GITHUB_IO_SYNC_ENABLEDunset or set to a value other than1. - Configure
TAMARAMACK_GITHUB_IO_SYNC_TOKENand theTAMARAMACK_GITHUB_IO_SYNC_ENABLEDrepository variable, keeping the variable non-1during the first validation if it is newly introduced. - Run Sync portfolio to tamaramack.github.io manually from Actions, selecting the desired source ref in GitHub’s workflow dispatch control (there is no separate ref input).
- Inspect the generated destination pull request, confirm that only
apps/portfolio/changed, allow destination checks to pass, and merge it according to destination policy. - Set or retain
TAMARAMACK_GITHUB_IO_SYNC_ENABLED=1for future sourcemainpushes.
Recovery:
- Stale machine pull request or branch: verify that the pull request is exactly
automation/portfolio-sync→developmentintamaramack/tamaramack.github.io, then rerun the source workflow manually. The run recreates the machine branch from current destinationdevelopmentand updates the open pull request. If the pull request metadata is wrong or cannot be updated, close only that verified machine pull request, delete only the verified destinationautomation/portfolio-syncbranch, and rerun to create a fresh pair. - Rejected path safety: do not bypass the failed check or widen the staged paths. Inspect the reported destination origin, base branch,
apps/profile/sentinel, symbolic-link checks, and staged paths; repair the destination layout or sync implementation that violated the contract, then rerun manually. - Rotated or revoked token: replace
TAMARAMACK_GITHUB_IO_SYNC_TOKEN, confirm its destination-only access and required permissions, and start a new manual run. A failed run does not resume with the replacement secret. - No destination PR checks: inspect the PR URL in the failed source run, confirm that the destination workflows and token-triggered PR events are enabled, then rerun manually. Do not treat a created PR without checks as verified.
- Destination layout change: first unset
TAMARAMACK_GITHUB_IO_SYNC_ENABLEDor set it to a non-1value. Coordinate changes to the source workflow, sync helper/tests/docs, and destination validation/promotion/deploy workflows while keepingapps/profile/destination-owned; validate with a manual run and destination pull request before restoringTAMARAMACK_GITHUB_IO_SYNC_ENABLED=1.
Site media in the GitHub.io copy
The site’s own media stays tracked in core/web/public/{i,v,d} (owner decision 2026-10-17), so the git archive snapshot copies it like any other tracked file and the copy serves it at the same /i, /v, /d paths; the default og:image is the tracked /i/portfolio/manhattan-2150-keyframe.png, absolute on the copy’s site URL (site-media.md). The snapshot carries no catalog media (m/ blobs never go in git), and the copy has no media server to ask for a URL, so catalog (new / admin) media would load from the production CDN:
core/web/shared/media-public.tsholdsPRODUCTION_MEDIA_BASE_URL.sitePublicMediaBase(useRuntimeConfig())returnsruntimeConfig.public.mediaBaseUrl(envNUXT_PUBLIC_MEDIA_BASE_URL, absolutehttp(s)only) when set, else that constant;sitePublicMediaUrl(key, config)joins it with a catalog key (m/pf/i/hero@640.webp). The module is client-safe and has no imports. The deployed app keeps using the server-sidemediaPublicBaseUrl()from@tgmc/media-nuxt.- The constant is empty (same origin) until the first production deploy. The follow-up commit right after that deploy sets it to the prod CloudFront origin (stack output
AssetsCdnUrl); from the next sync on, the copy requestshttps://<cloudfront>/m/pf/.... CloudFront serves those to anyone, and<img>/<video>need no CORS. - Site media never goes through it:
useSiteMediareturns public paths. Only catalog media rendered on the public site would.
Retired (outside four-env topology; files removed):
.github/workflows/ci-github-pages.yml— staging → GitHub Pages.github/workflows/cd.yml— generic artifact pack (superseded by CI local-parity + SAM CD).github/workflows/docker-build.yml— GHCR image publish (Compose is local-only)
CI skip rules (version-bump PRs)
ci.yml short-circuits (job still succeeds — required status checks stay green) when any of:
- Head branch in this repository starts with
ci/version-bump-(also for aworkflow_dispatchrun on that branch; a fork branch with that name gets full CI) - PR title contains
[skip ci]or[ci skip] - PR has label
skip-ci
Automated bump PRs use the ci/version-bump-* branch + skip-ci label. Do not put [skip ci] in the commit message: GitHub skips the whole workflow for that HEAD commit, so required checks never report and auto-merge stays blocked.
Why the bump workflows start CI themselves. A PR opened or updated with GITHUB_TOKEN gets pull_request runs that wait for approval (action_required, “Approve workflows to run”). While one is pending, GitHub ignores every other check on that commit — even a green dispatched Local parity — so the PR shows zero checks and stays BLOCKED (bump PRs used to sit like that until someone approved the run). Pushes and merges made with GITHUB_TOKEN start no push workflows. .github/actions/land-version-bump therefore:
- opens the bump PR, or reuses an open same-repository one for that branch (a closed PR from an earlier run, or a fork PR with the same branch name, does not count). The patch path (
yield-to-release) instead deletes its branch and stops when a release PR opened while it ran: the release bump supersedes it; - keeps one bump PR per base — the highest version wins (
sort -V): lower openci/version-bump-*PRs are closed as superseded, and the run closes its own PR when a higher one is open; - gets CI onto the bump head (
ensure_ci; Local parity short-circuits green there): waits up to 2 min for the head’spull_requestCI run and approves it when it isaction_required. If approval is refused, it deletes that pending run. It runsgh workflow run ci.yml --ref <bump branch>only when nopull_requestrun will report. Then it enablesgh pr merge --auto --squash --match-head-commit <pushed sha>; - waits up to 15 min for the merge. Behind the base → updates the branch, gets CI onto the new head the same way and re-arms auto-merge. Conflict (another bump landed first) → disables auto-merge and fails: re-run the workflow run, which bumps from the new base head and updates the same PR. Closed (superseded) → success. Timeout → disables auto-merge and fails;
- if a release PR (
<base>→main) is open, cancels its stalepull_requestCI and Regression gate runs, then getsci.ymlandregression-gate.ymlonto the new release head the same way (approve a pending run, else dispatch on<base>), soLocal parityandRegression gatereport there.
The bot merge itself starts no push workflows, so a version-only commit never re-runs regression, the dry-run or another bump.
Regression gate
Job Regression gate (regression-gate.yml → scripts/regression-gate.js) is the main ruleset’s second required check. It runs on PRs into main and on dispatches outside ci/version-bump-*, installs nothing and runs no tests:
- Release bump landed. It resolves the release level for
origin/main→ head (same range asversion-bump-release.yml) and fails unless the head version is at least main’s version bumped by that level — development patch bumps alone (1.6.2 for a 1.7.0 release) do not count. A release with no releasable commits needs no bump. - Regression green. It waits (polling every 30 s, ≤ 45 min) for the newest
Full regression (e2e)run ondevelopmentfor the head’s newest commit that is not a version bump, or for any verified bump commit above it (subjectchore: bump version toand onlypackage.json/package-lock.json/core/web/package.jsonchanged — a feature PR titled like a bump still needs its own run). When that commit is a release commit (Release-Source: <sha>trailer, see Release PR), the gate reads thedevelopmentcommit it names instead, and fails if the two trees are no longer identical (someone pushed to the release branch). It passes only when that run succeeded. Transient GitHub API errors are retried (5 in a row fail the check).
| Gate says | Do |
|---|---|
Release bump has not landed |
Wait for Version bump (release); if it failed, fix it and re-run it. It re-runs this check after landing the bump. |
Full regression (e2e) … is failure |
Fix development (new PR → merge → regression re-runs), or re-run the failed regression run if it was flaky; then re-run CI. |
No Full regression (e2e) run |
gh workflow run regression.yml --ref development, then re-run the gate once it is green. |
Release commit … no longer matches |
The release branch got a commit of its own. Land the change in development, then re-run npm run release:pr. |
still running after 45 min |
Re-run the gate after the regression run finishes. |
Waiting up to 45 minutes is expected right after a merge into development: the gate holds the release PR until that merge’s regression run finishes.
Automatic semver bumps
scripts/resolve-version-bump.mjs scans conventional commits and applies a channel policy:
| Channel | When | Policy |
|---|---|---|
development |
Push/merge to development |
Any releasable change → patch only; chore/docs/… → none |
main |
PR targeting main |
major if breaking; otherwise minor (patch elevated to minor) |
full |
Default / tests | Raw conventional level (patch/minor/major/none) |
Raw conventional mapping (before channel policy):
- none — only
chore/docs/ci/test/style - patch —
fix,perf,refactor,build,deps - minor —
feat - major —
BREAKING CHANGEfooter or!in the conventional header
Commit range: git log --first-parent <base>..<head> — each merged PR counts once, and histories merged in from elsewhere (the vendored Foundation history carries upstream BREAKING CHANGE: footers) cannot drive the bump. A GitHub merge commit (Merge pull request #N …) is classified by its PR title (first body line). Without first-parent, the 2026-10 release range read 18,148 commits instead of 313 and resolved a false major (2.0.0 instead of 1.7.0).
Release range (VERSION_BUMP_USE_BASE_VERSION=1, resolveReleaseBump): main is squash-only, so main..head reaches back past every earlier release. The current version comes from main, and the range starts at the head’s newest first-parent commit chore: bump version to <main version> — the bump that main last shipped (on development, the Version sync commit). Without that commit the whole main..head range counts. On a release branch the range is just the release commit, whose subject npm run release:pr derives from this same range on development.
development path (version-bump.yml): push ci/version-bump-{version} → PR into development → land action (above). Skipped while a legacy release PR (development → main) is open; a release/v* PR does not pause it, because its bump lands on the release branch.
main path (version-bump-release.yml): push ci/version-bump-{version} → PR into the release PR head (the release/v* branch) → land action, which then re-starts CI on the release head. Same-repo PRs only; rulesets block direct pushes to development/main.
Back to development (version-sync.yml): after the squash into main, the released version goes back to development as chore: bump version to <version> (release sync), so later patch bumps continue from it.
Not bumped: theme packages (@tgmc/theme*), packages/*, and web-e2e — they keep independent versions.
Loop prevention: GITHUB_TOKEN merges start no push runs; github-actions[bot] guards cover any other path (and version-bump.yml also skips chore: bump version to subjects); ci/version-bump-* + skip-ci label for CI short-circuit. Commit messages omit GitHub’s [skip ci] token so required workflows still run (needed for auto-merge).
How test CD works (no CI re-run)
cd-aws.yml on push to main:
- Job
deployable-gate—scripts/ci-is-deployable.mjs(Nx affected + path allowlist);workflow_dispatchforcesdeployable=true - Job
deploy-test—needs: deployable-gate,environment: test, runs whendeployable=true
Full lint/test/build already ran on the PR, and Full regression (e2e) ran after the merge to development. Require status checks on the main ruleset (Local parity, Regression gate) so a squash merge cannot land without green PR CI and a green post-merge regression. Squash creates a new SHA on main, so CD intentionally does not look up check-runs by commit.
Skip-CI trap on main squash
GitHub skips every workflow on a push when any commit message in that push contains [skip ci] / [ci skip] / [no ci] / [skip actions] / [actions skip].
main is squash-only. If the repository squash setting is Commit messages (COMMIT_MESSAGES), GitHub pastes every commit subject from the PR into the squash body. Old version-bump commits still say chore: bump version to … [skip ci], so a development → main release squash silently suppresses CD AWS Test (and Pages / Update AGENTS). That is why merges after mid-August 2026 produced no Actions runs on main.
Required repo setting (apply with npm run gh:setup-merge-settings):
| Field | Value |
|---|---|
squash_merge_commit_title |
PR_TITLE |
squash_merge_commit_message |
BLANK |
Never put GitHub’s [skip ci] token in commit subjects (version-bump already uses the skip-ci label + branch short-circuit instead). Keep release PR titles free of those tokens.
Manual recovery when CD was skipped: gh workflow run "CD AWS Test" --ref main.
Nx Cloud and affected CI
The private ubuntu-latest CI runner has 8 GB RAM. The development Nuxt build exceeded Node’s previous 4 GB and 6 GB heap caps in PR #386. The web app now keeps Nitro route chunks separate for the local/development node-server preset, reducing the size of the single Rollup bundle; the AWS Lambda preset retains its single bundle. CI keeps a 6 GB heap limit to leave memory for DynamoDB and the operating system.
CI uses Nx Cloud remote cache (nxCloudId in nx.json) and:
npx nx reset
npm run build:libs
npx nx affected -t test,build --parallel=3 --exclude=@tgmc/web
# when @tgmc/web is affected:
npm run build --workspace=@tgmc/theme
npx nx run @tgmc/web:test
npx nx run @tgmc/web:build
npx nx affected -t test:build-matrix --parallel=1
after nrwl/nx-set-shas (full git history via fetch-depth: 0) and, on PRs to development when web/docker are affected, the separate SYS_ENV=development Nuxt build. The nx reset + build:libs pair before Affected drops a poisoned in-job local cache left by that Development step; web is excluded from the parallel graph so a theme remote-cache restore cannot clear theme/core/dist mid-Nitro (see docs/solutions/build-errors/nx-local-cache-utilities-dist-enoent.md). The next step, Affected build matrix, runs test:build-matrix (today only @tgmc/nuxt-layer-ui): real nuxt builds of an all-features-on and an all-features-off fixture app, one at a time, about 45 s (nuxt-layer-ui.md § Development). It is not part of npm test or the pre-push hook. CI also runs node --test on scripts/ci-is-deployable.test.mjs, scripts/resolve-version-bump.test.mjs, scripts/cicd-workflow-policy.test.mjs, scripts/nitro-output-dir.test.mjs, scripts/e2e-regression.test.mjs, scripts/regression-gate.test.js, scripts/release-pr.test.js, scripts/sam-build.test.mjs, scripts/sam-deploy-flow.test.mjs, and scripts/sam-aws-helpers.test.mjs, and scripts/sam-template.test.js (parses infra/sam/template.yaml: Retain/PITR/versioning/CORS/throttle/domain), scripts/media-smoke.test.js (the media smoke script against a fake fetch), scripts/media-usage-rebuild.test.js (the usage rebuild script against a fake fetch), scripts/media-transfer.test.js (media:transfer argument parsing and exit codes), and scripts/lib/is-main.test.js / scripts/lib/aws-config.test.js (the shared script helpers). Lint excludes @tgmc/utilities, @tgmc/web-e2e, and @tgmc/media-player (pre-existing lint debt / vendor wrappers); their test/build still run when affected.
Repo secret (required): NX_CLOUD_ACCESS_TOKEN — create in Nx Cloud for this workspace and add under GitHub → Settings → Secrets and variables → Actions. CI fails fast if unset. Do not set NX_NO_CLOUD in workflows.
Deployable gate
scripts/ci-is-deployable.mjs sets deployable=true|false from:
- Affected Nx projects in the allowlist (
@tgmc/web, utilities, media-player, likwidlibs, theme packages, web-layer-admin,@tgmc/nuxt-layer-base,@tgmc/nuxt-layer-ui) - Or path changes under
infra/sam/**,docker/**,.github/workflows/**,scripts/sam-*.mjs,scripts/docker-compose.mjs,scripts/ci-is-deployable.mjs(+ its test), rootpackage.json/package-lock.json
Docs/AGENTS/prd-only changes → deployable=false → AWS Test dry-run and cd-aws.yml skip. workflow_dispatch always deploys test (then production dry-run). Production CD: GitHub Release published (non-prerelease) or manual cd-aws-prod.yml with confirm=deploy.
Dry-run matrix
| When | Workflow | Dry-run target | Deploy? |
|---|---|---|---|
Push / merge to development only |
cd-aws-test-dry-run.yml (one job) |
test (sam-dry-run.mjs test) |
No |
After successful CD AWS Test on main |
cd-aws.yml job sam-dry-run-production |
production (sam-dry-run.mjs production) |
No |
No dry-run in pull requests. Test dry-run must not run on PRs, main, or prod CD. Production dry-run must not run on development.
development branch — AWS Test dry-run only
- No GitHub Environment named
developmentfor Actions (historical GHCR workflow removed). - On push to
development, whendeployable=true, AWS Test SAM dry-run only (cd-aws-test-dry-run.yml→scripts/sam-dry-run.mjs test:sam validate+sam build+ local public→S3 sync plan) — nosam deploy, no deployment records. - Full CI does not re-run on push (it ran on the PR).
- Full regression (e2e) runs here, after the merge (
regression.yml) — the only place e2e runs automatically. Release PRs intomainread its result throughRegression gate; when it fails, fixdevelopmentbefore promoting. - Real deploys: test on
main(cd-aws.yml), production on Release published or manual (cd-aws-prod.yml). After a successful test deploy,cd-aws.ymlalso runs a production dry-run (build only).
E2E-only affection (@tgmc/web-e2e) does not force a deploy.
How production approval is enforced
- Release path —
cd-aws-prod.ymlruns onrelease: published(skips prereleases). Deploys the tagged commit (github.event.release.tag_name). Still requires GitHub Environmentproductionreviewers. - Manual path —
workflow_dispatchrequires inputconfirm=deploy - GitHub Environment
production— required reviewers (PROD_REVIEWERS),prevent_self_reviewwhen supported
Release PR
npm run release:pr (scripts/release-pr.js) opens the PR into main. Because main is squash-only, it never receives development’s commits, so a PR straight from development lists all of them (10,000+, including the vendored Foundation history) even when it changes a few dozen files. The release PR has one commit:
- Fetches
origin/mainandorigin/development. - Refuses when
mainchanged files after the last release thatdevelopmentdoes not match (a fix landed straight onmain): the release would revert them. Land the same change indevelopment, then re-run. The last releaseddevelopmentcommit comes from the newest release title onmain, else--source <sha>, else the head of the PR that mademain’s tip. - Resolves the release level like
version-bump-release.ymland builds a commit withgit commit-tree: tree =development, parent =main, subjectfeat(release): v<version> (development <sha>)(feat(release)!:for major,chore(release):when nothing is releasable), trailerRelease-Source: <full sha>. Signed whencommit.gpgSignis on. The working tree is never touched, so it runs from any checkout, uncommitted work included. - Force-pushes
release/v<version>, then opens the PR (or updates the open one); an open PR from anotherrelease/v*branch is closed as superseded.
Then the usual checks run: Version bump (release) lands the bump on the release branch, and the Regression gate follows Release-Source to the development e2e run. Squash-merge with the PR title (it names the source, which the next release reads). Version sync brings the version back to development.
npm run release:pr -- --dry-run builds the commit and prints the plan without pushing. Re-run the script after more work lands in development; it rewrites the release branch, and the bump is applied again. Never push commits to a release/v* branch by hand.
Recommended release flow
feature → PR → development (Local parity) → merge
→ Full regression (e2e) + CD AWS Test (dry-run) + patch bump PR (auto-merges)
→ npm run release:pr → PR release/v<version> → main (one commit; Local parity + Regression gate;
minor|major bump PR auto-merges into the release branch first) → squash merge
→ CD AWS Test (auto deploy) → on success production dry-run; Version sync PR into development
→ tag v{version} on main (match package.json) → GitHub Release (Publish)
→ CD AWS Production (Environment approval) → SAM portfolio-prod
Create the release from a commit on main that has already passed test CD. Tag name convention: v1.0.10 (root package.json version). Prereleases do not auto-deploy; use workflow_dispatch with confirm=deploy if needed.
Production checklist (first deploy and every release)
One-time setup in GitHub Settings → Environments → production (script: PROD_REVIEWERS=<user> npm run gh:setup-environments):
| Name | Kind | Required | Notes |
|---|---|---|---|
AWS_ROLE_ARN |
secret | yes | OIDC role trusted by repo:OWNER/REPO:environment:production (OIDC bootstrap) |
NUXT_ADMIN_TOKEN |
secret | yes | Bearer secret for admin + /api/media/*; empty fails writes closed |
MEDIA_PRESANITIZE_SECRET |
secret | recommended | Verifies the admin’s signed PDF / MP4 uploads; a different value from test; no spaces or quotes (Admin preSanitized secret). Required once the enforce step ships |
SITE_URL |
var | yes | Canonical origin; must be public https:// (never localhost — sam-cd-deploy rejects it for prod and test) |
CORS_ALLOW_ORIGIN |
var | yes | Exact front-door origin; not empty and not * (also the S3 CORS origin for uploads) |
DOMAIN_NAME |
var | optional | Custom domain for the HTTP API (e.g. www.example.com). Needs CERTIFICATE_ARN; the two are validated as a pair |
CERTIFICATE_ARN |
var | with domain | Regional ACM certificate in us-west-2 covering DOMAIN_NAME (must be ISSUED before deploy) |
HOSTED_ZONE_ID |
var | optional | Route 53 zone → stack creates the alias A record. Omit to add the CNAME yourself at an external registrar (target = DomainTarget stack output) |
API_THROTTLE_BURST |
var | optional | Positive integer, default 50 (HTTP API default-route burst) |
API_THROTTLE_RATE |
var | optional | Positive integer, default 25 (requests/second) |
MEDIA_UPLOAD_ORIGINS |
var | optional | Comma-separated exact origins (https://admin.example.com) allowed to PUT presigned uploads to the assets bucket, in addition to CORS_ALLOW_ORIGIN |
MEDIA_SMOKE_ENABLED |
var | optional | 1 runs the post-deploy media smoke (see below) after the greet check; unset or anything else skips it |
MEDIA_USAGE_REBUILD_ENABLED |
var | optional | 1 runs the post-deploy media usage rebuild (see below) right after the media smoke; unset or anything else skips it |
Set DOMAIN_NAME, CERTIFICATE_ARN and HOSTED_ZONE_ID only as GitHub Environment variables on production, never as repository-level variables. Repository vars are visible to the test Environment too, so portfolio-test and portfolio-prod would both try to claim the same custom domain (the second AWS::ApiGatewayV2::DomainName create fails, and a shared Route 53 alias would flip-flop). Give test its own distinct values if it should get a domain.
Blank optional vars are omitted from the deploy, so the template defaults apply and (with no domain vars) the stack creates no domain resources. Test (cd-aws.yml) receives the same optional vars from the test Environment.
Before the first deploy (the full ordered steps are in the production first-deploy runbook):
- Required reviewers on the
productionEnvironment (prevent_self_review); on plans without that feature, rely on theconfirm=deploymanual path and the Actions approval UI. - Request/validate the ACM certificate in us-west-2 (API Gateway regional domains use a certificate in the API’s own region, unlike CloudFront’s
us-east-1). - Confirm the
teststack deployed from the same commit (the release flow tags a commit that already passed test CD). - After deploy: create the DNS record (alias is automatic with
HOSTED_ZONE_ID; otherwise CNAME →DomainTarget), thencurl https://<domain>/api/greet. - Optional: set
MEDIA_SMOKE_ENABLED=1on the Environment.cd-aws.ymlandcd-aws-prod.ymlthen runscripts/media-smoke.jsright after the greet check, against the stack’sHttpApiUrlwith theNUXT_ADMIN_TOKENsecret: it presigns a 1x1 PNG upload, PUTs it, completes the ingest, lists it, waits for its on-ingest processing (the detail’sprocessingsteps; on AWS this proves the S3 upload trigger and the worker end to end, and a failed, stalled or not-started step fails the run), materializes thetthumbnail (polling the job), and deletes the asset. Every asset it creates is taggedsmokeand removed on failure too. Each request times out after 30 s (MEDIA_SMOKE_REQUEST_TIMEOUT_MSoverrides) so the step cannot hang the job; if an upload succeeded butcompletefailed, the failure names the orphaned object key and the next presign of that name reclaims it. Locally the same script runs againstdocker:testwithnpm run docker:test:smoke; against any stack:BASE_URL=https://… ADMIN_TOKEN=… npm run media:smoke. It is a verification probe of the deployed stack, not media processing, and the only media step a workflow may contain (No media work in GitHub Actions). - Optional: set
MEDIA_USAGE_REBUILD_ENABLED=1on the Environment.cd-aws.ymlandcd-aws-prod.ymlthen runscripts/media-usage-rebuild.jsright after the media smoke step (it runs whether or not the smoke is enabled), against the stack’sHttpApiUrlwith theNUXT_ADMIN_TOKENsecret:POST /api/media-usage/rebuildrebuilds the usage list (which pages use which assets) from every published post and every slot with an asset, and printsmedia usage rebuild: OK (added N, removed N, countersRepaired N). It exits 1 on a non-2xx answer (for example 503Media library unavailable) or a network failure, without printing the token; each request times out after 60 s. It is idempotent: recovery is to re-run the failed job or step, or press Rebuild usage list on the admin Media page (admin § Used by, refusal and Delete anyway). Locally:BASE_URL=https://… ADMIN_TOKEN=… node scripts/media-usage-rebuild.js. It reads the stored posts and slots and processes no media; with the smoke it is one of the two media-related steps a workflow may contain (No media work in GitHub Actions). Enable it ontestandproduction. On Lambda the web awaits each write’s live usage rows for at most 5 s before it answers (core/web/server/plugins/media-usage-wait.ts, site media § Usage list writes); rows whose write the bound cut staypending(deletes of those assets are refused) until this rebuild or the admin button settles them.
Release flow: the recommended release flow above — squash to main → test CD → tag v{version} → publish GitHub Release → approve the production Environment → portfolio-prod.
Data protection (what a bad deploy cannot destroy):
MessagesTable,PostsTable,MediaTableandAssetsBuckethaveDeletionPolicy: RetainandUpdateReplacePolicy: Retain: deleting the stack or replacing a resource leaves the data in place. The bucket name is fixed, so re-creating the stack while the old bucket exists fails until the old bucket is emptied/removed or imported (CloudFormation resource import).- All three tables have point-in-time recovery (35-day window).
AssetsBucketis versioned; noncurrent versions expire after 30 days, and delete markers with no remaining versions are removed (ExpiredObjectDeleteMarker). Every metadata repair (npm run sam:sync:repair-types, or the repair step after each sync) rewrites objects in place, which creates a new object version and keeps the previous one for 30 days. - The public-asset sync’s
--deletepass excludesm/*(catalog blobs),media/*(legacy prefix) and the site media dirsi/*,v/*,d/*, so a deploy never deletes media (No media work in GitHub Actions). -
Media URLs come from
MEDIA_PUBLIC_BASE_URL, set by the template to the stack’s ownAssetsDistributionon test and production (not a GitHub variable; the shared test CDN inNUXT_APP_CDN_URLhas nom/*). See infra.md § Media library. - Logs: Nitro and the media worker write to
/aws/lambda/<stack>-nitro-logsand/aws/lambda/<stack>-media-worker-logswith 30-day retention. These names differ from Lambda’s default/aws/lambda/<function>groups on purpose: on stacks that already ran (e.g.portfolio-test) the default group exists and a CloudFormationLogGroupwith the same name would fail withAlreadyExists. The old default group stays behind without retention; delete it by hand once the new group is receiving logs.
Rollback: runbook step 8 (CloudFormation, a previous release, table PITR, object versions).
Production first-deploy runbook
The owner runs these steps in order from a workstation, for the first portfolio-prod deploy and again for every release. Each step has its commands, the output to expect, and what to do when it fails; do not move to the next step until the current one passes. Commands are bash, run from any up-to-date checkout of this repo (they read origin/* and never touch the working tree). Region is us-west-2. <ver> is the release version (for example 1.7.0); <n> is a pull-request number.
Site media stays in core/web/public (/i, /v, /d) and ships with every deploy, so production needs no media import. Seeding the catalog (m/*) with npm run media:transfer is optional.
0. Preconditions
-
Tools and sign-in.
aws --version(v2),sam --version,gh --version,node --version(24). Then:aws sts get-caller-identity --query Account --output text # the production account id gh auth status # Logged in to github.com npm ci && npm run build:libs # only needed for media:transferIf it fails:
aws sso login/ setAWS_PROFILE;gh auth login. -
OIDC role. The account has the GitHub OIDC provider and a deploy role trusted by
repo:OWNER/REPO:environment:production(OIDC bootstrap). The preflight sees only that theAWS_ROLE_ARNsecret exists; it cannot read the role’s trust policy, so check the trust condition in IAM once. -
productionEnvironment. Create it with reviewers, then set the secrets and variables from the checklist table:PROD_REVIEWERS=<login>[,<second-login>] npm run gh:setup-environments gh secret set AWS_ROLE_ARN --env production # prompts for the value gh secret set NUXT_ADMIN_TOKEN --env production gh variable set SITE_URL --env production --body "https://<front-door host>" gh variable set CORS_ALLOW_ORIGIN --env production --body "https://<front-door host>" gh variable set DOMAIN_NAME --env production --body "<host>" # optional, with CERTIFICATE_ARN gh variable set CERTIFICATE_ARN --env production --body "arn:aws:acm:us-west-2:…" gh variable set HOSTED_ZONE_ID --env production --body "<zone id>" # optional gh secret list --env production && gh variable list --env productionExpected: the lists show
AWS_ROLE_ARN,NUXT_ADMIN_TOKEN(names only),SITE_URL,CORS_ALLOW_ORIGIN, and the domain variables you chose. Set the domain variables on the Environment, never at repository level.- Self-review deadlock: the script turns on
prevent_self_review, so the person who publishes the Release cannot approve its deployment. With one reviewer, either add a second reviewer, have that reviewer publish the Release, or untick Prevent self-review on the Environment in the GitHub UI. The preflight fails when the only reviewer is the signed-inghuser with self-review blocked; if the Environment lets admins bypass its rules (can_admins_bypass) it warns instead, because the only way through is a bypass that deploys with no review at all. - CORS: production uses the exact
SITE_URLorigin (scheme + host, no path, never*). CD fails closed on an empty or*value. -
SITE_URLwithout a custom domain: CD needs a publichttps://SITE_URLbefore the stack exists, and the API URL is only known after the first deploy. For the first deploy set both variables to the placeholder originhttps://portfolio-prod.invalid(.invalidis reserved by RFC 6761 and never resolves, so nobody can own it):gh variable set SITE_URL --env production --body "https://portfolio-prod.invalid" gh variable set CORS_ALLOW_ORIGIN --env production --body "https://portfolio-prod.invalid"Until the second deploy in step 6.2 points both at the stack’s
HttpApiUrl, the site’s canonical andog:urltags name the placeholder, the HTTP API CORS and the bucket’s S3 CORS (presigned admin uploads) allow only the placeholder origin, so browser uploads from the real origin fail. Do not announce the site before step 6.2. - No userinfo:
SITE_URLandCORS_ALLOW_ORIGINnever containuser:pass@; CD and the preflight reject it, and the preflight redacts it from its output.
- Self-review deadlock: the script turns on
-
Media smoke on test first.
gh variable set MEDIA_SMOKE_ENABLED --env test --body 1. Turn it on forproduction(same command with--env production) only after it passes on test in step 3. -
Test stack health before promoting. The media pipeline (#322) adds an S3 notification to the test bucket; CloudFormation replaces the bucket’s whole notification config, so any hand-made notification would be lost silently:
npm run prod:preflight -- --stack portfolio-test --skip-syncExpected:
OK: …withbucket-notificationeither empty or the managedm/→ media-queue rule.CORS_ALLOW_ORIGINon test may be*(a warning, not a failure). If it fails: fix the named check; for an unexpected notification, record it and decide with the owner before promoting.
1. Confirm development is green
gh run list --workflow regression.yml --branch development --limit 1 --json conclusion,headSha,url
gh run list --workflow cd-aws-test-dry-run.yml --branch development --limit 1 --json conclusion,headSha,url
git fetch origin && git rev-parse origin/development
Expected: both runs "conclusion": "success" on the headSha that equals origin/development. If it fails: fix on a feature branch → PR into development; never push to development or main.
2. Promote development → main
npm run release:pr -- --dry-run # Release release/v<ver>: minor (main <old> → <ver>) … Dry run: nothing pushed.
npm run release:pr # prints the release PR URL
gh pr checks <n> --watch # Local parity, Regression gate, version bump: all pass
gh pr merge <n> --squash # PR title only (it names the release source)
Expected: the dry run names the bump and versions; the PR has one commit; the checks pass and the bump commit lands on the release branch before the merge. If it fails: main already matches development means nothing to release. A red Regression gate means the development e2e run for that source failed; fix forward on development and re-run npm run release:pr (it rewrites the release branch). Never push to release/v*, main or development by hand.
3. Verify the test stack
gh run list --workflow cd-aws.yml --branch main --limit 1 --json databaseId,headSha,status
gh run watch <databaseId> --exit-status
git fetch origin && git rev-parse origin/main # must equal the run's headSha
Expected: CD AWS Test succeeds with SAM deploy (test) (including Post-deploy smoke (GET /api/greet + SITE_URL) and, with MEDIA_SMOKE_ENABLED=1, a log ending media smoke: OK) and SAM dry-run production (no deploy). A Version sync PR into development follows.
Then check the stack as a user:
TEST_URL=$(aws cloudformation describe-stacks --stack-name portfolio-test --region us-west-2 \
--query "Stacks[0].Outputs[?OutputKey=='HttpApiUrl'].OutputValue" --output text)
TEST_URL="${TEST_URL%/}"
curl -sfS "$TEST_URL/api/greet" # {"message":…}
npm run prod:preflight -- --stack portfolio-test --skip-sync # OK, bucket-notification = managed config
Open $TEST_URL in a browser: the home page renders, navigation works, page source has siteUrl / og:url / canonical on the public https:// origin (never localhost), and /admin/media lists the catalog. Remember the headSha: it is the commit you tag in step 5.
If it fails: read the failed step’s log (gh run view <databaseId> --log-failed). The deploy step’s CloudFormation errors: aws cloudformation describe-stack-events --stack-name portfolio-test --region us-west-2 --max-items 30. Do not release; fix forward through development. Known open item: images under /_ipx answer 403 on test (TODO); decide whether it blocks the release.
Optional cleanup on test: once /aws/lambda/portfolio-test-nitro-logs and …-media-worker-logs receive events, delete the old default groups (aws logs describe-log-groups --log-group-name-prefix /aws/lambda/portfolio-test --region us-west-2, then aws logs delete-log-group --log-group-name <old name>).
4. Preflight production
npm run prod:preflight # human report; add -- --json for a machine-readable summary
Read-only: it uses describe/get calls, an aws s3 sync --dryrun against the bucket, and gh api GETs (an allowlist refuses anything else before it runs), spawns aws and gh without a shell, and never prints secret values or URL credentials. --region / AWS_REGION, --environment, --stack, --repo and --public-dir (an existing directory) are validated first. Exit 0 = pass, 1 = a check failed (including aws CLI not found / gh not found), 2 = bad arguments. Expected on the first deploy:
prod preflight: portfolio-prod (us-west-2), GitHub OWNER/REPO environment production
first deploy: the stack does not exist yet
PASS aws-credentials account 000000000000, region us-west-2
PASS stack portfolio-prod does not exist: first deploy
PASS leftovers no tables, bucket, queues or log groups with the stack names
SKIP queue-sse first deploy: created by the deploy
SKIP bucket-notification first deploy: created by the deploy
SKIP bucket-versioning first deploy: created by the deploy
SKIP tables-pitr first deploy: created by the deploy
SKIP sync-dryrun first deploy: created by the deploy
PASS github-reviewers reviewers: <login>
PASS github-secrets AWS_ROLE_ARN, NUXT_ADMIN_TOKEN set (names only)
PASS github-variables CORS_ALLOW_ORIGIN https://…, domain …, MEDIA_SMOKE_ENABLED=1
PASS site-url SITE_URL https://… is https and resolves
OK: 7 passed, 0 warnings, 0 failed, 5 skipped
With the https://portfolio-prod.invalid placeholder (no custom domain), site-url is a WARN … does not resolve …; fine before the first deploy … and the verdict is OK: 6 passed, 1 warning, …. That warning is acceptable only before the first deploy; on a re-deploy it is a FAIL until step 6.2.
On a re-deploy, the stack checks run instead of leftovers: queue SSE, bucket notification (empty or exactly the managed config), versioning, PITR on all three tables, and the sync dry run (no deletes under m/, media/, i/, v/, d/). For an exact sync preview, stage the release build first (npm run sam:build -- production, from a checkout of the release commit); without it the check previews the --delete pass against an empty directory (worst case) and says so.
If it fails:
| Check | Fix |
|---|---|
aws-credentials |
Sign in to the production account (step 0). |
stack (ROLLBACK_COMPLETE, *_IN_PROGRESS) |
Wait for in-progress operations; a ROLLBACK_COMPLETE stack must be deleted first (step 8, failed first create). |
leftovers |
Retained tables or bucket from an earlier failed create. Empty ones: delete them. With data: keep them and import them into the stack. |
bucket-notification |
Someone added a notification by hand; CloudFormation will replace it. Move it into infra/sam/template.yaml or remove it. |
sync-dryrun |
The sync would delete protected media. Do not deploy; fix the excludes in scripts/sam-aws-helpers.mjs (PUBLIC_ASSET_PROTECTED_PREFIXES). |
github-*, site-url |
Set the missing reviewer, secret or variable (step 0). A site-url warning before the first deploy is fine when the deploy or a DNS record creates the host. |
5. Publish the Release
gh release create v<ver> --target <headSha from step 3> --title v<ver> --generate-notes
gh run list --workflow cd-aws-prod.yml --limit 1 --json databaseId,status,headBranch
Expected: the release is published (not draft, not prerelease) on the commit that passed test CD, and CD AWS Production shows Waiting for the production review. Approve it in Actions → the run → Review deployments (as a reviewer who did not publish, if self-review is blocked). Then gh run watch <databaseId> --exit-status.
If it fails: a prerelease or draft does not deploy; edit the release or run gh workflow run cd-aws-prod.yml --ref v<ver> -f confirm=deploy. A wrong tag target: delete the release and tag (gh release delete v<ver> --cleanup-tag) and recreate. A failed deploy: step 8.
6. Verify production
Run 6.0 → 6.3 in order: the SITE_URL checks only pass once 6.1 / 6.2 are done.
6.0 The stack is up.
aws cloudformation describe-stacks --stack-name portfolio-prod --region us-west-2 \
--query "Stacks[0].[StackStatus,Outputs[].[OutputKey,OutputValue]]" --output text
API_URL=<HttpApiUrl output without the trailing slash>
curl -sfS "$API_URL/api/greet" # {"message":…}
curl -sSI "<AssetsCdnUrl output>/favicon.ico" | head -n 1 # HTTP/2 200 from CloudFront
Expected: CREATE_COMPLETE (first deploy) or UPDATE_COMPLETE; the outputs include HttpApiUrl, the three table names, MediaQueueUrl, AssetsBucketName, AssetsCdnUrl, AssetsDistributionId (and DomainTarget with a domain); the run log shows the greet smoke and, if enabled, media smoke: OK.
6.1 DNS (custom domain): with HOSTED_ZONE_ID the alias record is created; otherwise add a CNAME from DOMAIN_NAME to the DomainTarget output, then curl -sfS https://<domain>/api/greet.
6.2 SITE_URL second pass (no custom domain, first deploy only): replace the placeholder and redeploy the same tag:
gh variable set SITE_URL --env production --body "$API_URL"
gh variable set CORS_ALLOW_ORIGIN --env production --body "$API_URL"
gh workflow run cd-aws-prod.yml --ref v<ver> -f confirm=deploy # then approve, and gh run watch
Until this pass, the preflight’s site-url check fails (the placeholder never resolves, and the stack now exists); every other check must already pass.
6.3 Verify as a user, with PROD_URL = https://<DOMAIN_NAME> or $API_URL:
curl -sS "$PROD_URL/" | grep -oE 'siteUrl"?:"[^"]*"' # siteUrl:"<PROD_URL>", never localhost or the placeholder
npm run prod:preflight # every check PASS (re-deploy checks)
IPX=$(curl -sS "$PROD_URL/work/spatial-experiences" | grep -oE '/_ipx/[^"'"'"' ,]+' | head -n 1 | sed 's/&/\&/g')
if [ -z "$IPX" ]; then
echo "FAIL: no /_ipx image found on /work/spatial-experiences" # never curl "$PROD_URL" alone: the home page would answer 200
else
echo "$IPX" # a NuxtImg URL, e.g. /_ipx/…/i/portfolio/tesseract/….png
curl -sS -o /dev/null -L -w '%{http_code} %{content_type} %{url_effective}\n' "$PROD_URL$IPX" # 200 image/…
fi
Expected: siteUrl is the public origin; the preflight ends OK: … 0 failed; the /_ipx image answers 200 with an image/* type (after any redirect to the CDN). FAIL: no /_ipx image found is a defect too: the page no longer renders NuxtImg images (or the grep missed them), so check the page source before releasing. In a browser: home, every nav item, a case study (its images load), /docs, and /admin/media sign-in; console without errors; og:url and canonical on the public origin.
A 403 (or any 4xx/5xx) on the /_ipx image is a defect. The same 403 is open on test (TODO: deployed test images). The owner fixes it before announcing the release, or explicitly accepts it in writing (in the release notes or the TODO entry), per the repo rule that no error status is “expected” without the owner’s say-so.
6.4 Follow-ups:
- Media base URL: set
PRODUCTION_MEDIA_BASE_URLto theAssetsCdnUrloutput in a follow-up PR (TODO). - Optional: seed the catalog with
npm run media:transfer.
If it fails: siteUrl on localhost, the placeholder or the wrong host means SITE_URL is wrong; fix the variable and redeploy the tag. Lambda errors: aws logs tail /aws/lambda/portfolio-prod-nitro-logs --since 30m --region us-west-2 (media: …-media-worker-logs). Anything worse: step 8.
7. After every release
Merge the Version sync PR into development when its checks pass, and update the TODO with anything found in steps 3–6.
8. Roll back
Tables and the assets bucket are Retain with PITR and versioning (data protection), so none of these steps loses data.
-
Failed first create (
ROLLBACK_COMPLETE):aws cloudformation describe-stack-events --stack-name portfolio-prod --region us-west-2 \ --query "StackEvents[?contains(ResourceStatus,'FAILED')].[LogicalResourceId,ResourceStatusReason]" --output table aws cloudformation delete-stack --stack-name portfolio-prod --region us-west-2 aws cloudformation wait stack-delete-complete --stack-name portfolio-prod --region us-west-2 npm run prod:preflightThe first failed reason is the cause. The retained tables and bucket survive the delete and block the next create:
leftoverslists them; delete only the empty ones (aws dynamodb delete-table --table-name … --region us-west-2;aws s3api delete-bucket --bucket …, which fails while any object version remains), fix the cause, and publish again (or dispatchconfirm=deployon the same tag). -
Failed update: CloudFormation rolls back to the previous template by itself (
UPDATE_ROLLBACK_COMPLETE); the old code keeps serving. Read the events as above and fix forward. If the stack sticks inUPDATE_ROLLBACK_FAILED, fix the named resource and runaws cloudformation continue-update-rollback --stack-name portfolio-prod --region us-west-2(add--resources-to-skip <LogicalId>only for a resource you have checked by hand). -
Bad code that deployed: redeploy the last good release tag:
gh workflow run cd-aws-prod.yml --ref v<previous> -f confirm=deploy, then approve. Only use a tag that already ran onportfolio-prod; an older template would remove resources added since. The first production release has no predecessor, so fix forward with a patch release instead. Re-publishing the previous Release (edit it to a draft, then publish it again) fires the samerelease: publisheddeploy of its tag; the dispatch does the same with less churn. -
Bad table data (PITR, 35 days, never in place):
aws dynamodb describe-continuous-backups --table-name portfolio-prod-Posts --region us-west-2 \ --query "ContinuousBackupsDescription.PointInTimeRecoveryDescription.[EarliestRestorableDateTime,LatestRestorableDateTime]" aws dynamodb restore-table-to-point-in-time --source-table-name portfolio-prod-Posts \ --target-table-name portfolio-prod-Posts-restore --restore-date-time 2026-10-18T17:00:00Z --region us-west-2 aws dynamodb wait table-exists --table-name portfolio-prod-Posts-restore --region us-west-2Check the restored table (
aws dynamodb scan --table-name portfolio-prod-Posts-restore --select COUNT --region us-west-2, orget-itemon a known key). The restore copy has no PITR, TTL or tags, which is fine for a scratch copy. Then pick one of the two ways back:A. Copy items back into the live table (no downtime; the stack keeps its fixed table names).
putreplaces an item with the same key; items created after the restore point stay, so delete those by hand if they must go. To copy only some items, add--filter-expression/--expression-attribute-valuesto the scan.SRC=portfolio-prod-Posts-restore DST=portfolio-prod-Posts aws dynamodb scan --table-name "$SRC" --region us-west-2 --output json > restored-items.json # the CLI pages the scan itself node -e ' const fs = require("fs"); const [dst] = process.argv.slice(1); const { Items } = JSON.parse(fs.readFileSync("restored-items.json", "utf8")); for (let i = 0; i < Items.length; i += 25) { const batch = { [dst]: Items.slice(i, i + 25).map((Item) => ({ PutRequest: { Item } })) }; fs.writeFileSync(`batch-${String(i / 25).padStart(5, "0")}.json`, JSON.stringify(batch)); } console.log(`${Items.length} items in ${Math.ceil(Items.length / 25)} batches`); ' "$DST" for f in batch-*.json; do aws dynamodb batch-write-item --request-items "file://$f" --region us-west-2 --output json > "$f.out" || break node -e 'const u = JSON.parse(require("fs").readFileSync(process.argv[1], "utf8")).UnprocessedItems || {}; if (Object.keys(u).length) { console.error(`unprocessed items: re-run ${process.argv[1]}`); process.exit(1); }' "$f.out" || break doneExpected:
N items in M batches, then nounprocessed itemsmessage. If one appears, the loop stops there: re-run that batch file (aws dynamodb batch-write-item --request-items file://batch-0000N.json …) until it is clean, then re-run the wholeforloop from the top, so the batches after it are written too. Repeating batches is safe: every write is an idempotent put of the same restored item. Delete thebatch-*,restored-items.jsonfiles and the restore table afterwards (aws dynamodb delete-table --table-name "$SRC" --region us-west-2).B. Replace the whole table under its live name (downtime: the site errors between the delete and the end of the restore). A restore cannot overwrite a table, so back up the verified restore copy and the live table, wait until both backups are
AVAILABLE(DynamoDB refuses to delete a table while its backup is in progress, and restores only from an available backup), delete the live table, and restore the restore-copy backup under the live name. SetLIVE/RESTOREfor the table you are replacing:LIVE=portfolio-prod-Posts RESTORE=portfolio-prod-Posts-restore REGION=us-west-2 wait_backup() { # $1 = backup ARN; polls every 10 s for up to 30 min [ -n "$1" ] || { echo "no backup ARN (create-backup failed?)" >&2; return 1; } for _ in $(seq 1 180); do status=$(aws dynamodb describe-backup --backup-arn "$1" --region "$REGION" \ --query BackupDescription.BackupDetails.BackupStatus --output text) || return 1 echo "backup $1: $status" [ "$status" = AVAILABLE ] && return 0 [ "$status" = DELETED ] && { echo "backup $1 was deleted" >&2; return 1; } sleep 10 done echo "timed out after 30 min waiting for backup $1 to be AVAILABLE: stop, do not delete $LIVE" >&2 return 1 } STAMP=$(date -u +%Y%m%dT%H%M) RESTORE_ARN=$(aws dynamodb create-backup --table-name "$RESTORE" --backup-name "$RESTORE-$STAMP" \ --region "$REGION" --query BackupDetails.BackupArn --output text) SAFETY_ARN=$(aws dynamodb create-backup --table-name "$LIVE" --backup-name "$LIVE-before-replace-$STAMP" \ --region "$REGION" --query BackupDetails.BackupArn --output text) echo "restore copy: $RESTORE_ARN"; echo "safety (old live data): $SAFETY_ARN" wait_backup "$RESTORE_ARN" && wait_backup "$SAFETY_ARN" \ && aws dynamodb delete-table --table-name "$LIVE" --region "$REGION" --query TableDescription.TableStatus \ && aws dynamodb wait table-not-exists --table-name "$LIVE" --region "$REGION" \ && aws dynamodb restore-table-from-backup --target-table-name "$LIVE" --backup-arn "$RESTORE_ARN" \ --region "$REGION" --query TableDescription.TableStatus \ && aws dynamodb wait table-exists --table-name "$LIVE" --region "$REGION" \ && aws dynamodb update-continuous-backups --table-name "$LIVE" --region "$REGION" \ --point-in-time-recovery-specification PointInTimeRecoveryEnabled=true \ && echo "OK: $LIVE replaced from $RESTORE_ARN, PITR re-enabled"Expected:
backup …: CREATINGlines, thenAVAILABLEfor both ARNs,"DELETING","CREATING", andOK: … PITR re-enabled. The&&chain stops at the first failure, so nothing is deleted unless both backups are available. If it stops after the delete, do not delete or restore again: ifwait table-not-existstimed out (a large table deletes slowly), re-run just that waiter and then the chain from therestore-table-from-backupline; ifwait table-existstimed out (a large table restores slowly), re-run just that waiter and then theupdate-continuous-backupsline; if the restore itself failed, re-run therestore-table-from-backupline with the same$RESTORE_ARN(the old live data is in$SAFETY_ARN).Then re-apply the per-table settings a restore drops. A restored table keeps no PITR, TTL or tags, and CloudFormation does not notice (stack drift). PITR is part of the chain above for every table. Every table also gets its template tags back (
SYS_ENV,App), using the table ARN fromdescribe-table:TABLE_ARN=$(aws dynamodb describe-table --table-name "$LIVE" --region "$REGION" --query Table.TableArn --output text) aws dynamodb tag-resource --resource-arn "$TABLE_ARN" --region "$REGION" \ --tags Key=SYS_ENV,Value=production Key=App,Value=portfolio aws dynamodb list-tags-of-resource --resource-arn "$TABLE_ARN" --region "$REGION" # SYS_ENV=production, App=portfolioTTL is needed only where the template defines it:
Table Re-apply after a restore portfolio-prod-MessagesPITR (in the chain) portfolio-prod-PostsPITR (in the chain) portfolio-prod-MediaPITR (in the chain), then TTL: aws dynamodb update-time-to-live --table-name portfolio-prod-Media --region us-west-2 --time-to-live-specification Enabled=true,AttributeName=ttlFinish with
npm run prod:preflight: itstables-pitrcheck fails if PITR was missed. Keep$SAFETY_ARNuntil the site is verified; on-demand backups never expire, so delete it by hand afterwards (aws dynamodb delete-backup --backup-arn "$SAFETY_ARN" --region us-west-2), along with the restore-copy backup and the-restoretable. -
Overwritten or deleted object (versioning, 30 days):
aws s3api list-object-versions --bucket <AssetsBucketName> --prefix <key> --region us-west-2 \ --query "{versions: Versions[].[VersionId,IsLatest,LastModified], markers: DeleteMarkers[].[VersionId,IsLatest]}" aws s3api delete-object --bucket <AssetsBucketName> --key <key> --version-id <delete-marker id> # undelete aws s3api copy-object --bucket <AssetsBucketName> --key <key> \ --copy-source "<AssetsBucketName>/<key>?versionId=<good version id>" # restore an overwriteUnder
m/(catalog media):copy-objectwrites a new object, so it fires the bucket’ss3:ObjectCreated→ media-queue trigger. The worker skips it when the asset’s catalog row is more than 5 minutes older than the event, or when there is no row (…was registered before this upload event…; skipped/…has no catalog row or reservation; skippedin/aws/lambda/portfolio-prod-media-worker-logs), so nothing is reprocessed.- Removing a delete marker fires no
ObjectCreatedevent. - Either way, a restored object only serves when the Media table still has its catalog row (and the child rows for its variants). Restore missing rows first, from PITR as above; otherwise the object is an orphan that admin cannot list or delete.
-
Stack deleted by mistake: the tables and bucket remain. Re-create the stack by importing them, or rename them out of the way before a fresh deploy; never delete a bucket or table that holds production data.
Promote media to production
Catalog media is not deployed by CD (the sync excludes m/* and media/*). To seed a stack’s catalog from a local one, run npm run media:transfer from a workstation. Swap --from and --to to pull a backup. The tool copies the catalog rows (DynamoDB) and the objects (S3) with the same ids, names and paths, so media URLs are identical in every environment. Rules: media.md § Transfer between backends.
Before you start
npm ci && npm run build:libs: the CLI loads@tgmc/media,@tgmc/media-awsand@tgmc/media-postgresfrom their builtdist.- AWS credentials for the target account in the shell:
AWS_PROFILE, orAWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY/AWS_SESSION_TOKEN. The region comes from--region, thenAWS_REGION, elseus-west-2. - Permissions on the media table:
dynamodb:GetItem,PutItem,DeleteItem(releasing a name hold after a failure),Query,TransactWriteItemsandConditionCheckItem(a child insert is a transaction with a condition check on its asset, and IAM checks each action in it). - Permissions on the assets bucket:
s3:GetObject,PutObjectandDeleteObjectonm/*, pluss3:ListBucket. WithoutListBucket, S3 answers 403 instead of 404 for a missing key, and every asset fails. - The table and bucket names come from the stack outputs
MediaTableNameandAssetsBucketName:
aws cloudformation describe-stacks --stack-name portfolio-prod --region us-west-2 \
--query "Stacks[0].Outputs[?OutputKey=='MediaTableName' || OutputKey=='AssetsBucketName'].[OutputKey,OutputValue]" --output text
Which local catalog. sqlite:<dir> is a catalog folder holding media.sqlite and blobs/. Each app uses MEDIA_LOCAL_ROOT when it is set, otherwise:
| App | Run as | Catalog folder |
|---|---|---|
Admin (core/admin) |
Docker admin Compose (host ./data mounted at /app/data) |
data/admin-media |
| Admin | nuxt dev in core/admin |
core/admin/data/admin-media (<cwd>/data/admin-media) |
Web (core/web) |
npm run dev (Nuxt runs in core/web) |
core/web/.data/media (<cwd>/.data/media) |
| Web | npm run docker:local |
data/media |
| Web | npm run docker:dev with a non-DynamoDB backend |
named volume tgmc-dev-media-data (copy it to a folder first) |
Endpoints. sqlite:<dir> (above), aws:<table>@<bucket> (a stack’s media table and assets bucket), and postgres:<postgres://… URL> for a PostgreSQL catalog with its objects in --blobs fs:<dir> or --blobs s3:<bucket> (--from-blobs / --to-blobs when both sides are Postgres; --s3-endpoint / --path-style for MinIO). A Postgres source must already have the media tables; a real Postgres target gets them on open, a dry-run one is never created. Passwords in the URL are masked in all output, but prefer PGPASSWORD so they stay out of shell history. Details: media-postgres § Transfer endpoint.
Promote. Every step has a gate: do not go on until it passes.
SRC=sqlite:./data/admin-media # the catalog folder from the table above
DST=aws:portfolio-prod-Media@<AssetsBucketName>
# 0. Stop the app that owns the source catalog, then back the folder up. Opening a SQLite
# source runs its schema migrations in place, so keep an untouched copy.
mkdir -p ./backups && cp -a "${SRC#sqlite:}" "./backups/$(basename "${SRC#sqlite:}")-$(date +%Y%m%d-%H%M)"
# 1. Dry run: reads both sides, writes nothing.
npm run -s media:transfer -- --from "$SRC" --to "$DST" --dry-run > transfer-dry-run.json; echo "exit $?"
- Gate 1: the dry run must exit
0. On exit1,conflictsorfailed(top level or underchildren) is not empty. Stop, resolve every entry (see below), then run the dry run again. Repeat until it exits0andsourceCountmatches the number of assets you expect to promote. A dry run opens a SQLite target read-only, so an existing SQLite target must already be on the current schema: on an older schema every asset shows asfailedand trips this gate (open it once with the app, or run a real transfer, to migrate it). - During steps 2 and 3, nobody uploads through the admin to the target stack. The transfer holds each name on the target while it copies that asset, and writes objects create-only. A clash therefore ends up as a conflict rather than lost data, but you want a quiet target so that the dry run you approved is the run you get.
# 2. Real run: same arguments, without --dry-run. Keep the report.
npm run -s media:transfer -- --from "$SRC" --to "$DST" > transfer.json; echo "exit $?"
# 3. Smoke the stack's media API.
BASE_URL=https://<domain> ADMIN_TOKEN=<NUXT_ADMIN_TOKEN> npm run media:smoke
- Gate 2: the real run must exit
0. On1, read the report, fix the cause, and rerun. Reruns are safe: what already landed is skipped. - Gate 3:
media:smokeprintsmedia smoke: OK.
Backup pull (production into a new local catalog; also good for checking a promotion). Use read-only credentials: the pull never writes to its source, and read-only credentials make that a guarantee. The pull needs dynamodb:GetItem and dynamodb:Query on the media table (and its indexes), and s3:GetObject on m/* plus s3:ListBucket on the bucket.
npm run -s media:transfer -- --from aws:portfolio-prod-Media@<AssetsBucketName> --to "sqlite:./backups/media-$(date +%Y%m%d)"
Add --tenant <name> (repeatable: portfolio, example-organization) to pull only that tenant’s media, for example --to sqlite:./backups/ha --tenant example-organization; without it every tenant is copied (media.md § Transfer between backends).
A dry run never creates or migrates a sqlite: target: a missing target reads as empty, so everything is reported as would-copy.
Reading the report. stdout carries only the JSON report; library log lines go to stderr. Use npm run -s so npm’s banner stays out of the file. The report’s top level lists asset ids and children lists child ids, and sourceCount is the number of source assets seen. The exit code is 0 when nothing conflicted or failed, 1 otherwise (the report is still printed, even after a listing error), and 2 for bad arguments (with usage).
skipped: the target already has that id with the samesha256and its object (checked for existence, not re-hashed), so reruns are safe. A rerun also copies ready children the target is missing.conflictsandconflictReasons: nothing was overwritten. Causes:- the same id with other bytes or another path
- the name held by another asset or by an upload in progress (also when the name was taken during the run)
- different bytes already at the key, or an object that appeared at the key during the run
- a child id or path already in use
Resolve by hand: rename or delete one side, then rerun.
failed: a missing or corrupt source object, a target write that failed the re-hash, an asset or child already on the target whose object is missing there, or an AWS error. Alisting <tenant> (offset <n>)entry means the source listing broke off, so the rest of that tenant was not attempted. Fix the cause and rerun. What a failed item leaves on the target:- Usually nothing the run wrote: no new row, no new object, and the asset’s name hold released.
- An asset or child whose object is missing: the existing row stays. Restore the object or delete the row.
- An error mentioning
re-reading the row failed: the insert’s outcome is unknown, so the verified object is kept (the row may have landed). A rerun adopts it. - An
audit: …entry: the asset and children were copied and stay; only theasset.transferredevent is missing. A rerun skips them and does not retry the event.
- Jobs and audit history stay behind. Each asset the run wrote gets one
asset.transferredaudit event on the target. - Only
readychildren are copied. Pending or failed variants can be materialized again on the target.
Local rehearsal: npm run docker:test, then run the transfer against aws:portfolio-test-Media@portfolio-assets --dynamodb-endpoint http://127.0.0.1:8000 --s3-endpoint http://127.0.0.1:4420 --path-style with AWS_ACCESS_KEY_ID=minioadmin AWS_SECRET_ACCESS_KEY=minioadmin, then npm run docker:test:smoke.
Local
cp .env.example .env # SYS_ENV=local
npm run db:migrate:local
npm run dev # or npm run build
Development (Docker — local only)
Docker Compose is for local DynamoDB Local + MinIO + web/api stacks (Postgres opt-in) and Lambda test parity. Compose images are not built or used on GitHub Actions. The one container CI does run is a services: sidecar: amazon/dynamodb-local:2.5.2 in the local-parity job (below). Full regression (e2e) after merge to development (regression.yml) uses npm run e2e:regression: SYS_ENV=test identity, NITRO_PRESET=node-server, SQLite stores (migrated, plus one published fixture post from scripts/e2e-seed-posts.js — test data only, so the /blog/<slug> route is audited), Cypress on :4200, server entry .output/test/server/index.mjs. The suite checks the current primary navigation, every portfolio route at four viewport sizes, and the blog API store response. Do not set job-level NODE_ENV=production before npm ci — that omits root devDependencies (e.g. @nx/vite) and breaks postinstall → nuxt prepare.
cp .env.development.example .env.development
npm run docker:build:dev # ensureNitroOutput(development) then web + api images
npm run docker:dev # DynamoDB Local + MinIO + api (:4100) + web (:4200); exclusive with docker:local on :4200
npm run docker:dev:down
npm run docker:local # DynamoDB Local + MinIO (DATA_BACKEND=sqlite for ./data); exclusive with docker:dev on :4200
npm run docker:local:down
# Full regression (same path as CI — no Docker required)
npm run e2e:regression
The regression runner requires a successful HTTP response (2xx) before starting Cypress. A server returning 500 stays unready and the timeout reports its last HTTP status, rather than producing a failure for every browser test.
The web and admin Nitro configurations bundle nuxt so Nuxt 4.6 renderer imports resolve their generated manifest/precomputed data instead of the package stubs. They also bundle @primeuix/utils so PrimeVue’s utility barrel and subpath exports resolve together; externalizing only the subpaths can produce missing-export errors when the built server starts. Validate these settings with the built-server regression command above, not only a successful build. Admin missed this when the lockfile moved to Nuxt 4.6 and returned 500 on every page in Docker; core/web/tests/nitro-externals.spec.ts and core/admin/tests/nitro-externals.spec.ts guard both inline sets.
Optional hot-reload profile (does not replace the built stack):
docker compose -f docker/docker-compose.dev.yml --project-directory . --profile hotreload up
Build images locally with npm run docker:build:dev (docker/Dockerfile.app with Compose args for api :4100 / web :4200). See docker/README.md.
Test / Production (SAM)
Canonical AWS region: us-west-2 (samconfig, CD workflows, .env.*.example).
npm run sam:build # SYS_ENV=test → stage + sam build (fails hard unless SAM_BUILD_SOFT=1)
npm run sam:deploy:test
node scripts/sam-build.mjs production
npm run sam:deploy:prod
The SAM build launches Nuxt with a 6 GiB Node heap limit. Its server bundle can exceed Node’s default 2 GiB heap even when the workflow sets NODE_OPTIONS for the job.
Local machines without the SAM CLI can set SAM_BUILD_SOFT=1 to stage Nitro output and continue; CI/CD must never set this. Windows: install SAM via winget install -e --id Amazon.SAM-CLI (or the 64-bit MSI); enable long paths; see infra/sam/README.md.
Static assets (CloudFront + private S3)
Test (portfolio-test): hashed assets publish to the shared Example Organization/portfolio CDN (shared-cdn-test stack → s3://shared-cdn-test-assets-000000000000/ + https://cdn.example.com). Bucket root only (no portfolio/ prefix). Sync never uses --delete (would remove Example Organization objects; publisher IAM also omits s3:DeleteObject). Do not invalidate /* on that distribution. Stack-owned AssetsBucket / AssetsDistribution remain until cutover is fully verified; Lambda NUXT_*_CDN_URL imports shared-cdn-test-url.
Production (portfolio-prod): exclusive private S3 + CloudFront on the portfolio stack; sync uses --delete.
CD (scripts/sam-cd-deploy.mjs) and local npm run sam:deploy:*:
- Resolves
NUXT_APP_CDN_URL: GitHub var → else shared-cdn-test export (test) / stackAssetsCdnUrl(prod) → else empty on pass 1 - Builds Nuxt with that CDN (if known), deploys the stack, then on first deploy rebuilds when CDN was unknown and redeploys
- Syncs
infra/sam/.aws-sam/build-src/public→ the CDN bucket root (no prefix). Test: shared bucket without--delete. Prod: stack assets bucket with--delete. Source is Nitro.output/publicfromscripts/sam-build.mjs. Every sync/repair pass excludesm/*(catalog blobs,m/{tenant}/{kind}/{name}.{ext}), the legacymedia/*prefix andpending/*(presigned uploads waiting forcomplete), so CD never uploads, deletes or rewrites catalog media; the--deletepass and the repair also exclude the site media dirsi/*,v/*,d/*(their git-tracked files always upload through the typed passes; that split is permanent), so--deletecan never remove media (No media work in GitHub Actions). Sync sets explicitContent-Typeper extension (.js→text/javascript,.css→text/css,.mp4,.pdf,.html/.htm, …) andCache-Control(public, max-age=31536000, immutableonly for hashed_nuxt/**;public, max-age=3600, must-revalidatefor everything else, including the unhashed_nuxt/builds/*manifest), and then REWRITES object metadata so identical-byte re-syncs cannot leave AWS CLI’sbinary/octet-streamdefault in place (fatal under CloudFrontnosnifffor module scripts). On the shared test bucket the repair is limited to hashed_nuxt/*assets, so HyperActivity’s objects outside_nuxt/keep their existing headers; the exclusive prod bucket is repaired everywhere outside the media prefixes (m/,media/,pending/,i/,v/,d/). - Lambda runtime receives the same CDN URL from the template (
NUXT_APP_CDN_URL/NUXT_PUBLIC_CDN_URL)
Local sync plan: npm run sam:sync:dryrun or node scripts/sam-sync-public.mjs --dryrun --shared. Heal existing wrong MIME without a rebuild: npm run sam:sync:repair-types (shared test CDN; never --delete). After a metadata-only heal, path-scoped CloudFront invalidation /_nuxt/* may be needed — never invalidate /* on the shared distribution. Docker docker:test does not emulate S3/CDN.
Set GitHub Environment test vars NUXT_APP_CDN_URL / NUXT_PUBLIC_CDN_URL to https://cdn.example.com so builds skip bootstrap guessing. Production still uses stack AssetsCdnUrl until a shared-cdn-production cutover is requested.
Test SITE_URL: set Environment test var SITE_URL to the stack HttpApiUrl (e.g. https://….execute-api.us-west-2.amazonaws.com, no trailing slash). cd-aws.yml no longer falls back to http://localhost:4200. When the var is empty or still localhost, scripts/sam-cd-deploy.mjs reads HttpApiUrl from the existing portfolio-test stack; if that is also missing (first-ever create), CD fails closed. Post-deploy smoke asserts the HTML siteUrl / og tags are not localhost and match HttpApiUrl. Background: aws-test-site-url-localhost.
Post-deploy smoke: CD curls GET {HttpApiUrl}/api/greet (HttpApiUrl uses the HTTP API $default stage — no /test or /prod path prefix).
No media work in GitHub Actions
Media never goes through a workflow: nothing in Actions ingests, enriches or transforms images, video or documents, and nothing touches the catalog (m/, media/). The only media files CD copies are the site’s own git-tracked ones under core/web/public/{i,v,d} (below).
- Sync excludes. In
scripts/sam-aws-helpers.mjs,MEDIA_PIPELINE_PREFIXESism/*,media/*,pending/*(presigned uploads waiting forcomplete: a production--deletewould otherwise wipe uploads in flight) andLEGACY_SITE_MEDIA_PREFIXESisi/*,v/*,d/*;PUBLIC_ASSET_PROTECTED_PREFIXESis all six. The typed per-extension passes exclude the media pipeline prefixes; the catch-all--deletepass and the metadata repair exclude all six (after their includes; AWS CLI filters: last match wins). The dry-run upload plan (npm run sam:sync:dryrun,sam:dry-run) lists exactly the keys some pass uploads.scripts/sam-aws-helpers.test.mjsreplays the filters against sample keys. - Site
/i,/v,/dobjects (permanent). Site media stays tracked incore/web/public/{i,v,d}and is served from there (owner decision 2026-10-17; plan PR 5a’s untracking is cancelled). The typed passes upload those tracked files with theirContent-Typeand the shortCache-Controlon every deploy, including the site’s images, videos, PDFs andd/htmlabs. Files with an unmapped extension there (i/_derived/.gitkeep) are not uploaded. The keys are not write-protected: changing a tracked file replaces its object and metadata through its typed pass. Only deletion and the metadata repair are blocked:--deletenever removes an object underi/,v/ord/, and the repair never rewrites one, so removing a file from git leaves its object in the bucket until someone deletes it deliberately. This typed sync (option (b)) replaces plan PR 5b’s “every pass excludesi/,v/,d/” for good. - Workflow guard.
scripts/cicd-workflow-policy.test.mjsscans every file in.github/workflows/and fails onmedia:*orcontent:enrich-imagesscripts, direct calls of their script files, thetgmc-mediaCLI,/api/image/*and/api/media/*calls, and anynpm/pnpm/yarnstep that names a root script resolving to any of them through a chain of aliases. Script names are matched as tokens rather than by parsingnpm runsyntax, so every option form is covered (run-script,--silent,--prefix .,--workspace .). The one allowed step is theMEDIA_SMOKE_ENABLED-gated post-deploy media smoke incd-aws.yml/cd-aws-prod.yml, and only its single exact command lineBASE_URL="${API_URL%/}" node scripts/media-smoke.js(no arguments, no second call, no other base URL): a verification probe against the deployed stack that creates onesmokeasset and deletes it, not media processing on the runner. - E2E. Cypress answers every
GETwhose path has an/m/segment (any origin), directly (/m/...) or through Nuxt Image (/_ipx/<modifiers>/m/...), with the 92-byte fixturecore/web-e2e/src/fixtures/media-placeholder.webp(core/web-e2e/src/support/media.ts, registered for every spec), so CI never needs the real media.ui-audit.cy.tsfails on images that finished loading without pixels; the placeholder passes. - Promoting media is a workstation job: Promote media to production.
Media storage deploy notes
What the admin / media hardening (2026-10) changes on a SAM deploy. One sam deploy carries the template and the code together; CloudFormation updates the IAM role and the bucket before the new function code serves traffic, so there is no manual ordering beyond deploying test before production.
- Versioned media URLs.
AssetsDistributiongains anm/*cache behaviour with the newMediaCachePolicy(thevquery string in the cache key). EverypublicUrl/previewUrlnow ends in?v=<sha8>(media § Versioned public URLs). URLs copied before the deploy (nov) keep working: they are cached as the no-query version. Distribution updates take a few minutes to propagate; nothing to do by hand. - Private pending prefix. Presigned uploads now land under
pending/(bucket policy deny for CloudFront,ExpirePendingUploadslifecycle rule, Nitro IAM grant; infra § Media library). In-flight uploads: a presigned URL handed out before the deploy still PUTs to the publicm/key; itscompleteafter the deploy finds no pending object and registers the public one as before (sanitized in place, CDN purged), so it is not lost. Nothing to migrate; existing assets never move. An abandoned pre-deploy upload left at itsm/key is still removed by the next presign of that name (blob.stray_removed). After the deploy, check that a test upload’s object appears underpending/only untilcomplete(aws s3 ls s3://<AssetsBucket>/pending/ --recursive), and thathttps://<AssetsDistribution>/pending/...answers 403. - Admin token at build. The deploy step now passes the secret as
ADMIN_TOKEN_SECRETand the build never sees it (above). Nothing to configure: the GitHub secret keeps its nameNUXT_ADMIN_TOKEN. A build made before this change may still carry the old token in.output; the deploy replaces it, and rotating the secret afterwards retires the baked copy. These workflow changes could not be run here (Actions was rate-limited); check the first CD run’s deploy log forAdminToken=***and that the admin still signs in. - Admin token strength. The web (admin APIs and the
/api/media/*admin principal) now refuses a weakNUXT_ADMIN_TOKENlike a missing one: under 32 characters or a placeholder. Before deploying, make sure thetestandproductionsecrets are long random values (openssl rand -hex 32); otherwise the admin and the post-deploy media smoke get 401 until the secret is replaced and CD runs again. Update every.env.admin(MEDIA_API_TOKEN/ADMIN_TOKEN) that forwards to that stack at the same time.
Admin token
GitHub Environment secret NUXT_ADMIN_TOKEN reaches the Lambda as SAM parameter AdminToken → Lambda env NUXT_ADMIN_TOKEN / ADMIN_TOKEN, and only that way: it is runtime-only (review M5). cd-aws.yml / cd-aws-prod.yml pass it to the deploy step as ADMIN_TOKEN_SECRET (never as NUXT_ADMIN_TOKEN, which the Nuxt build reads); scripts/sam-cd-deploy.mjs puts it into the parameters and then removes ADMIN_TOKEN_SECRET / NUXT_ADMIN_TOKEN / ADMIN_TOKEN from the environment before the build runs, and core/web/nuxt.config.ts sets runtimeConfig.adminToken: '' (the server reads the env, resolveAdminToken). So .output never contains the token. scripts/cicd-workflow-policy.test.mjs fails if a deploy step sets NUXT_ADMIN_TOKEN or ADMIN_TOKEN. Empty = admin writes fail closed. It must be at least 32 characters and not a placeholder (openssl rand -hex 32): the server refuses a weak token like an empty one (401 with the reason), so check the secret before deploying this change (environment § ADMIN_TOKEN). npm run prod:preflight only sees that the secret exists, not its value.
Admin preSanitized secret
GitHub Environment secret MEDIA_PRESANITIZE_SECRET (and MEDIA_PRESANITIZE_SECRET_PREVIOUS during a rotation) on test and production. cd-aws.yml / cd-aws-prod.yml pass both to scripts/sam-cd-deploy.mjs, which sends them as the NoEcho SAM parameters MediaPresanitizeSecret / MediaPresanitizeSecretPrevious → Nitro Lambda env of the same names. The web verifies with them the HMAC the admin puts on the PDF and MP4 uploads it already sanitized (media-nuxt § Admin preSanitized signature). For now (expand phase) nothing is refused while either side lacks the secret; the web logs presanitize.no_key / presanitize.unsigned. A later step (Phase 7 S1-13) refuses unsigned admin PDF / MP4 uploads on Lambda and makes CD fail without the secret.
- Values. A different value on
testandproduction, so a test secret can never sign for production. Generate each without spaces or quotes, e.g.openssl rand -hex 32(64 characters): it travels inside the space-delimited--parameter-overridesstring, andsam-cd-deployrefuses whitespace or quotes (naming the variable, never printing it).redactCliArgsmasks both parameters in error messages. - Deploy. A blank
MEDIA_PRESANITIZE_SECRETis left out, so the stack keeps its value.MediaPresanitizeSecretPreviousis always sent, as""when blank: the SAM CLI drops an unquotedKey=, which would keep the old key throughUsePreviousValue. - Rollout (web first, test first). Create the secret on
testand let CD deploy; then onproductionbefore the next release. Then put the same value in that target’s.env.admin(one file per target) and rebuild the Docker admin (docker § Docker-only admin); an admin that goes first only causespresanitize.no_key. Upload a PDF and an MP4 through the admin: Logs Insights on the Nitro log group (filter evt like "presanitize." | stats count() by evt) showspresanitize.verifiedand none ofpresanitize.unsigned,presanitize.no_key,presanitize.bad_signature. -
Rotation (test first). Deploy the web with
MEDIA_PRESANITIZE_SECRET_PREVIOUS= the old value andMEDIA_PRESANITIZE_SECRET= the new one; rebuild the admin with the new value; once Logs Insights showspresanitize.verifiedfrom the new admin and nopresanitize.bad_signature, clear_PREVIOUSand deploy again (it sendsMediaPresanitizeSecretPrevious="", so the old key stops verifying). Check it without printing a value, and never run these queries withoutlength(…):aws lambda get-function-configuration --function-name <NitroFunction name> --query "length(Environment.Variables.MEDIA_PRESANITIZE_SECRET_PREVIOUS)" # → 0 aws lambda get-function-configuration --function-name <NitroFunction name> --query "length(Environment.Variables.MEDIA_PRESANITIZE_SECRET)" # → greater than 0 (64 for openssl rand -hex 32)
CORS
SAM parameter CorsAllowOrigin (default *). For production, set GitHub Environment var CORS_ALLOW_ORIGIN to the front-door origin(s) you serve from (single origin string). Leaving * in prod widens blast radius if a browser client ever carries Authorization.
The assets bucket’s S3 CORS uses the same origin (plus MediaUploadOrigins when set), allows GET/HEAD/PUT only (PUT is for presigned media uploads; POST is not allowed) and only the headers the presigned PUT needs (Content-Type, If-None-Match, Cache-Control, Content-Disposition, Content-MD5, x-amz-*; no bare * — scripts/sam-template.test.js checks the list against PRESIGN_REQUIRED_HEADERS in packages/media-aws/src/presign-headers.ts). The test stack still passes *, so its bucket stays open until CORS_ALLOW_ORIGIN is set for that Environment.
Details: infra/sam/README.md.
GitHub Environments + Rulesets setup
# Environments (test + production)
PROD_REVIEWERS=your-github-user npm run gh:setup-environments
# Branch Rulesets (PR-only development/main; print checklist, optional apply)
npm run gh:setup-rulesets
APPLY_RULESETS=1 npm run gh:setup-rulesets
Environments
Creates/updates:
test— deployment branch policy formaingithub-pages— add deployment branchesdevelopmentandmain(environment is GitHub-managed; script does not PUT it)production— required reviewers (PROD_REVIEWERS),prevent_self_review, no branch deploys
Required reviewers need a GitHub plan that supports Environment protection rules (Team/Enterprise for private repos). On plans that return HTTP 422 for reviewers, keep the production branch/tag policy and gate prod with workflow approval in the Actions UI (and/or confirm=deploy on workflow_dispatch). The release: published path still runs under the production Environment.
Rulesets (branch protection)
Scripts: scripts/setup-github-rulesets.mjs. Recommended:
| Ruleset | Refs | Key rules |
|---|---|---|
devenv |
refs/heads/develop* |
Require PR; block force-push/delete; required check Local parity |
main |
refs/heads/main |
Require PR; squash only; conversation resolution; checks Local parity + Regression gate |
Admin/RepositoryRole bypass remains available for emergencies. If APPLY_RULESETS=1 returns 403/422, apply the checklist printed by the script in Settings → Rules → Rulesets.
Also create label skip-ci (the ruleset script attempts this) for version-bump PR detection.
Secrets / vars (per environment)
Repository-level (Actions):
| Name | Where | Notes |
|---|---|---|
NX_CLOUD_ACCESS_TOKEN |
Repository Actions secret | Nx Cloud access token for CI remote cache |
Prefer OIDC (no long-lived keys) for AWS deploy roles:
| Name | Where | Notes |
|---|---|---|
AWS_ROLE_ARN |
Environment secret | IAM role trusted by repo:OWNER/REPO:environment:NAME |
AWS_REGION |
Environment variable (preferred) or secret | Default us-west-2 if unset |
NUXT_ADMIN_TOKEN |
Environment secret | Blog admin Bearer secret (test + production) |
MEDIA_PRESANITIZE_SECRET / MEDIA_PRESANITIZE_SECRET_PREVIOUS |
Environment secrets (recommended; _PREVIOUS only during a rotation) |
Admin preSanitized HMAC keys, different per Environment (Admin preSanitized secret) |
OPENAI_API_KEY |
Environment secret (optional) | Required only when AI_LAB_LIVE_ENABLED=1 |
AI_LAB_SIGNING_SECRET |
Environment secret (optional) | HMAC for AI Lab continuation tokens; required with live mode |
NUXT_APP_CDN_URL |
Environment variable | Test: https://cdn.example.com (shared-cdn-test). Prod: stack AssetsCdnUrl or override. |
CORS_ALLOW_ORIGIN |
Environment variable (optional) | HTTP API CORS origin; default *. Set for production. |
DOMAIN_NAME / CERTIFICATE_ARN / HOSTED_ZONE_ID / API_THROTTLE_BURST / API_THROTTLE_RATE / MEDIA_UPLOAD_ORIGINS / MEDIA_SMOKE_ENABLED |
Environment variables (optional) | See the production checklist. |
OIDC bootstrap (account once)
- Create the GitHub OIDC provider:
https://token.actions.githubusercontent.com(client IDsts.amazonaws.com). - Create two roles with
sts:AssumeRoleWithWebIdentity, conditioned on:repo:OWNER/REPO:environment:testrepo:OWNER/REPO:environment:production
- Attach a deploy policy covering CloudFormation, SAM-managed services (Lambda, API Gateway, S3, DynamoDB, CloudFront, Logs), and IAM role management for
CAPABILITY_IAM. - Set each Environment secret
AWS_ROLE_ARNto the matching role ARN. CD workflows already requestid-token: write.
Trust policy sketch for the deploy role:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": { "Federated": "arn:aws:iam::ACCOUNT:oidc-provider/token.actions.githubusercontent.com" },
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
},
"StringLike": {
"token.actions.githubusercontent.com:sub": "repo:OWNER/REPO:environment:test"
}
}
}
]
}
Use separate roles (or separate sub conditions) for environment:test and environment:production.
Node version
Floor: Node.js 24+. Local and tooling may use Node 24 through Current. Deploy/CI stay on 24 for Lambda and image parity.
| Layer | Pin |
|---|---|
GitHub Actions (ci, cd-*, version-bump, update-agents) |
node-version: 24 |
Docker (Dockerfile.app / dev / production) |
node:24-bookworm-slim |
| SAM Lambda | nodejs24.x |
| Local tip | .nvmrc → 24 (or any >=24) |
package.json engines + .npmrc |
>=24 + engine-strict=true |
Node 20 is not supported. Installs fail under engine-strict if the runtime is below 24.
Why CI/Docker/Lambda stay on 24: Node 24 is Active LTS and available as nodejs24.x on Lambda. Newer Current releases (e.g. 26) are fine locally when engines allows >=24, but are not an AWS Lambda managed runtime yet. Node 20 Lambda runtime reached deprecation Phase 1 on 2026-04-30.
Nitro’s aws_lambda handler is async — compatible with Lambda’s Node 24 removal of callback-style handlers.