@tgmc/media
Node / Nitro / Lambda media catalog library for TGMC portfolio apps.
Package README: packages/media/README.md.
Role
| Concern | Owner |
|---|---|
| Asset inventory (image / svg / audio / video / model / document) | This package |
| Child recipes + materialize | This package |
| Image pixel transforms | @tgmc/image (composed) |
| SVG / audio / video / model / document transforms | Placeholders (TransformerNotImplementedError) until later |
| Local catalog / jobs / audit | SQLite adapter |
| Local blobs | FsBlobStore (filesystem, behaves like the S3 bucket) |
| AWS DynamoDB / S3 / Lambda | @tgmc/media-aws + @tgmc/media-nuxt /api/media/* |
| PostgreSQL catalog / jobs / audit | @tgmc/media-postgres |
| Admin grid / multi-upload UI | @tgmc/admin /admin/media + /api/admin/media/* |
| Nuxt / Next host adapters | @tgmc/media-client + @tgmc/media-nuxt |
Requirements: docs/plans/2026-09-22-001-feat-media-library-sdk-plan.md · Admin UI: docs/plans/2026-09-22-003-feat-admin-media-ui-plan.md · Hosts: docs/plans/2026-09-22-004-feat-media-host-adapters-plan.md.
API surface
import { createMediaLibrary } from '@tgmc/media';
const { library, close } = createMediaLibrary({
dbPath: './data/media.sqlite',
blobDir: './data/media-blobs',
});
// A host's local mode: `<root>/media.sqlite` + `<root>/blobs` (`localMediaBlobDir(root)`), on-ingest recipes in the request.
// const { library, close } = createLocalMediaLibrary({ root: './.data/media', sanitize });
await library.ingest({
kind: 'image',
data: './a.png',
mime: 'image/png',
meta: { tags: ['hero', 'logo'] },
});
await library.addAssetTags(assetId, ['brand']);
await library.listAssets({ tag: 'hero' });
await library.materializeChild(assetId, null, null, { variant: '640' }); // catalog variant
await library.materializeChild(assetId, 'square', {
kind: 'image',
ops: [{ type: 'resize', ops: { width: 800, height: 800 } }],
format: 'webp',
}); // custom recipe
await library.deleteAsset(assetId); // hard cascade; frees the name once the objects are gone; MediaInUseError (409 in_use) while a published page uses it
await library.deleteAsset(assetId, { force: true }); // admin: deletes an in-use asset, removes its usage rows, audits asset.force_deleted
await library.getAssetUsage(assetId); // the published pages that use it
await library.replacePageUsage(
{ tenant: 'portfolio', pageType: 'post', pageId },
{ assetIds, pageLabel, url, state: 'live' }
);
await library.rebuildUsage({ tenant: 'portfolio', pageTypes: ['post'], pages }); // full rebuild of those page types
close();
Scoped library
HTTP hosts never hand MediaLibrary to a route. scopeMediaLibrary(library, principal, { writesFrozen }) (src/scoped-library.ts) wraps it for one caller (MediaPrincipal: { type: 'admin' } or { type: 'tenant', tenant, tokenId, scopes }); @tgmc/media-nuxt builds it per request (media-nuxt § Auth).
AdminMediaLibrarypasses every call through unchanged. It also has the admin-only operations:getJob,readOriginal,materializeChild,enqueueMaterializeChild,failMaterializeDispatch,enqueueCatalogVariantIfMissing,runMaterializeJob,replacePageUsage,rebuildUsage(usage writes and the rebuild are admin-only; plan 3 writes them from trusted server code). It also takesdeleteAsset(id, { force })andgetAssetUsage(id). An adminlistAssetswithout atenantstill lists Portfolio only.TenantMediaLibraryhas none of the admin-only operations (a type test pins this) and pins every call to its tenant. Another tenant’s asset, child, job list or key reads as not found (MediaNotFoundError, 404), checked before anything is changed.reserveName,listAssetsandingestforce the tenant.registerRemoteAssetandingestforcepreSanitized: falseandstrictRaster: true.listJobsneeds anassetId(otherwiseMediaForbiddenError('admin_only')).resumeRenamechecks ownership first, so a tenant never resumes another tenant’s rename.deleteAsset(id, { force }):forceis admin-only, a tenant’s isMediaForbiddenError('admin_only')(403) before the freeze check and before any read, so nothing is read, audited or deleted; withoutforcea tenant deletes its own asset (an in-use one isMediaInUseError).getAssetUsage(id)checks the asset’s tenant first (another tenant’s asset is the same 404 as a missing id, before any usage read).- Tenant upload allowlist.
TENANT_UPLOAD_ALLOWLIST(src/tenant-allowlist.ts): a tenant may upload the five raster types (image/png,image/jpeg,image/webp,image/avif,image/gif) and SVG (image/svg+xml), nothing else.assertTenantUploadAllowed({ kind, mime, filename, route })checks the kind and both the sent MIME and the filename extension’s MIME; the route decides the kinds (presign: rasters,svg: SVG,ingest: either). The tenant library runs it inassertUploadSanitizable(which takeskindon both scoped types; the admin one drops it),reserveName(so nothing is reserved),registerRemoteAssetandingest, before the unscoped call. A refusal isMediaKindNotAllowedError→ 415kind_not_allowed(data: { code, kind, mime }). The admin keeps the kinds table (unsupported_type). strictRaster.ingest,registerRemoteAsset,sanitizeIngestBytesandsanitizeStoredUploadtakestrictRaster: a raster whose bytes no sniffer recognises is aMediaUploadInvalidError(415invalid_upload, the stored object deleted and its name released) instead of being kept as is. Without a sanitizer configured,strictRasteris a 503 (MediaSanitizeUnavailableError), never a silent pass-through. Unlike the other refusals, that 503 keeps the pending object and its reservation (a misconfigured host, not a bad upload); the reservation TTL cleans them up. The admin keeps today’s pass-through.ScopedAsset. Assets returned by a scoped library are frozen copies carrying a module-private brand.TenantMediaLibrary.listChildrentakes aScopedAsset(no second catalog read) or an id; a plainMediaAssetdoesn’t type-check.MediaLibrary.listChildrenitself also accepts a loaded asset or an id.- Freeze hook. Both scoped types have
assertWritable(), which throwsMediaWritesFrozenError(503media_frozen) whilewritesFrozen()is true. Every write calls it first (includingcompleteRegisteredUpload, which clears a tag), and so do the queue-sending helpers in@tgmc/media-nuxt(materializeOrEnqueue,rerunOnIngest). When frozen,resumeRenamereturnsfalseinstead of throwing, so a read still answers. The hook is off (() => false) until S2-1 wires it toMEDIA_WRITES_FROZEN. - Errors.
MediaForbiddenError(code)maps to 403data: { code }(admin_only,insufficient_scope,tenant_mismatch,pre_sanitized_forbidden) andMediaKindNotAllowedError(kind, mime)to 415kind_not_allowed, throughmediaErrorToHttp.
Keys, names and variants
The key is the URL. The same string is the CDN path, the MinIO/S3 key, the host-dev URL and the storage key (no leading slash on keys):
m/{tenant}/{kind}/{name}.{ext} original
m/{tenant}/{kind}/{name}@{variant}.{ext} processed variant
| Part | Values |
|---|---|
tenant |
pf portfolio (default), ha example-organization (TENANT_CODES) |
kind |
i image, s svg, v video, a audio, d document, o model (KIND_CODES) |
name |
lowercase kebab-case; the whole filename (name.ext) is at most 36 characters |
variant |
t (320 px), 640, 1280, 1920, or c-{8 hex} for a custom recipe; p (poster) is reserved |
@tgmc/media exports originalKey, variantKey, parseMediaKey (paths.ts), normalizeMediaFilename (meta-policy.ts), slugMediaFilename (filename.ts) and the variant helpers (variants.ts). MediaAsset carries tenant, name and path; MediaChild carries variant and path. The asset id stays an internal DB key and never appears in a URL.
Filenames
slugMediaFilename is mechanical: basename only, accents stripped, lowercase kebab-case ASCII, extension at most 8 alphanumerics, truncated so name.ext is at most 36 characters, untitled when nothing is left. The SQLite migration backfill uses it so existing names never change.
normalizeMediaFilename is the policy name every ingest uses (ingest, reserveName, registerRemoteAsset): it first drops generator prefixes (vimg, mj, midjourney, dall-e, firefly, sora, chatgpt), camera prefixes (IMG_, DSC_, PXL_ …), UUIDs, long hex tokens, digit runs of 10 or more and single-letter middle initials, then slugs the rest (falling back to the plain slug when nothing is left). Tamara G Mack_Resume_2026.pdf → Tamara Mack-resume-2026.pdf; vimg-tesseract-framework.mp4 → tesseract-framework.mp4.
IngestInput takes optional filename and tenant; without a filename the name comes from the file path (when data is a path), else {kind}-{first 8 of the sha256}.{ext}. Only an explicit mime is checked against the extension; without one the extension decides the mime, and an extensionless name takes the extension of the resolved mime.
Metadata policy
validateMediaMeta(kind, meta, { computed? }) → { ok, issues, meta } (assertMediaMeta throws RecipeValidationError instead). Rules (MEDIA_META_LIMITS): title required, 3–80 characters; alt at most 150 and required for image and svg (and video with a posterId); description at most 300; credit at most 80, defaulting to DEFAULT_MEDIA_CREDIT (Tamara Mack). Text fields must be human text, not identifiers: UUIDs, long hex, camera names, generator prefixes (vimg; mj only before _ / - or alone, since MJ is also initials), vendor job ids and filenames are rejected. width / height / durationSec are computed fields and are rejected unless computed: true (the site import sets it).
MediaLibrary checks the policy on ingest (before anything is reserved or written) and registerRemoteAsset. It is on by default: every admin and /api/media upload must carry a title and, for images and SVG, alt text or decorative: true. A failure is a MediaMetaValidationError, which mediaErrorToHttp sends as 400 data: { code: 'invalid_meta', issues: [{ field, code, message }] } (@tgmc/media-client throws MediaInvalidMetaError with the issues). enforceMetaPolicy: false (createMediaLibrary({ enforceMetaPolicy: false })) is the explicit opt-out for bulk tools that import files without editorial metadata: the bin/media.mjs CLI uses it. transferMedia copies rows with insertAsset and never runs the policy; the site import checks its manifest instead.
Edits. updateAssetMeta(id, patch, { validate: true }) is a person’s edit, used by both PATCH routes (admin and /api/media, whose { meta?, tags? } body becomes the patch; tags replace). The patch is merged over the stored meta and checked with validateMediaMetaEdit(kind, current, patch): width / height / durationSec in the patch are rejected (only the on-ingest recipes measure them), and the editorial rules run on the merged meta only when the patch touches an editorial field (EDITORIAL_META_FIELDS), so a tags-only edit of an older asset without a title still saves. A blank text field clears it; a blank credit falls back to the default. The stored meta is the normalized result. Internal writers (the on-ingest recipes) leave validate off.
Variants
- Catalog variants (
t/640/1280/1920): one child per (asset, variant, format). Output is webp by default;webp,avif,jpegandpngare selectable, and the format is told apart by extension (user@example.comnext touser@example.com;jpeguses.jpg). Re-materializing an existing (variant, format) replaces the object at the same key and reuses the child; thehooks.afterReplacehook runs so a CDN can drop the stale copy (createAwsMediaLibrarywires it to CloudFront invalidation whenASSETS_DISTRIBUTION_IDis set). A recipe that is exactly a width-only resize to 320/640/1280/1920 with no quality override is recognised as the catalog variant. - Custom recipes:
variant: 'custom', keyed@c-{first 8 hex of the child id}, a fresh child each time, never overwritten. roleis a free label (defaults to the variant code).pis rejected with a validation error. Posters will be separate image assets linked throughmeta.posterId.- Because keys have no attempt suffix, a retry or a stale worker writes the same key. The library never deletes a variant key on a lost lease or when replacing a live object; it deletes only when the asset is gone or after a successful requeue by the sole owner.
Materialize job state for rejected inputs
A source the image guards refuse (ImageInputError: over MAX_BYTES, an edge over MAX_EDGE_PX, or undecodable bytes) is a terminal failure, like a bad recipe: runMaterializeJob marks the job failed (lastError holds the message), marks the child failed (a live ready child keeps serving its old object), writes the child.materialize_failed audit event, and then rethrows the ImageInputError. HTTP hosts therefore answer 413 / 422 through mediaErrorToHttp with the real message, and the SQS worker (mediaWorkerHandler) acks it instead of redelivering (a redelivery of a failed job returns early anyway). Nothing is left queued, so the next “Create variant” for the same variant and format is accepted and fails the same way (422), never 409. Other terminal failures (RecipeValidationError, missing transformer) still return { child, job } with status failed. Retryable adapter errors still re-queue the job. On-ingest recipes (below) report a variant refused this way as failed, not retryable.
On-ingest recipes
runOnIngestRecipes(library, assetId, options?) (on-ingest.ts) is what every new asset gets automatically. It is idempotent, so re-running it (a redelivered S3 event, “Re-run processing”) changes nothing that is already done.
- Meta, read from the original through
library.readOriginal(asset)(a stat check against the ingest size cap first). Images getwidth,height,lqipanddominantColor(@tgmc/image); SVG getswidth/heightfrom the attributes or theviewBox(readSvgDimensions); a PDF document getspageCountwhen the optionalpdf-libpackage is installed (options.pdfPageCountinjects a reader). Only changed keys are written, so editorial meta stays. - Variants (
onIngestVariants): images get the catalogt,640,1280and1920(webp), never wider than the original (talways). SVG, video, audio, models and documents get none; a video poster stays a separate image asset linked throughmeta.posterId.
A ready variant is skipped, a failed one is made again, and one with an active job is left alone. A variant whose materialize fails ends as a failed child; one that hit an adapter fault (a thrown error) is also marked failed (failMaterializeDispatch, so it is not left queued with nothing to redeliver it) and reported retryable. The other variants still run. An original that cannot be read (missing object, storage fault) returns retryable with nothing written. An original that cannot be decoded, or is over the size cap, records meta.processingError and makes no variant; a later successful run clears it. The function never throws for a recipe failure, only MediaNotFoundError for a missing (or concurrently deleted) asset.
MediaLibrary takes hooks.onIngested (also through createMediaLibrary({ hooks }) and createAwsMediaLibrary({ onIngested })). It runs after ingest and registerRemoteAsset; an error is audited (asset.on_ingest_failed) and never fails the ingest, and the returned asset is re-read so it carries the new meta. onIngestRecipesHook is the hook that runs the recipes. onIngestStatus(asset, children, jobs, now?) gives each step’s status (queued / running / ready / failed, with the error) for the admin drawer. A step still waiting after ON_INGEST_STALL_MS (15 minutes, above the AWS worker’s give-up point of three 360 s receives) reads not-started (no child yet, counted from the asset’s createdAt) or stalled (its queued job, or its running job’s startedAt, is that old). The admin stops polling those and offers Re-run processing; scripts/media-smoke.js fails on them at once. Which runner calls the recipes depends on the host: see media-nuxt § On-ingest processing.
Concurrent runners. The S3 trigger, a duplicated S3 event and Re-run processing can run the recipes for the same asset at the same time; each variant still ends with exactly one child:
- A catalog child’s id is deterministic: a UUIDv5 of (asset, variant, extension) (
catalogChildId,child-ids.ts). An existing child keeps its id. - Child inserts are create-only. SQLite’s primary-key conflict, DynamoDB’s child
Put(attribute_not_exists(pk), in both transactions) and Postgres (an id check under the parent lock, with23505as the backstop) surface asMediaConflictError, which the runner reports as in progress, not failed. The shared catalog contract checks it on every adapter. - The runner enqueues through
library.enqueueCatalogVariantIfMissing(assetId, variant), which re-reads the children and returnsnullfor a ready one, so a late or duplicate run never re-materializes it or invalidates the CDN. - A failed attempt never deletes a key that a ready child serves or that another live job for the same child writes (
materialize-job.ts). It reads the jobs before the children, so a job that finishes between the two reads shows up as its ready child. - A queued job older than
ON_INGEST_STALL_MSis abandoned: nothing delivers an on-ingest job, so one orphaned between insert and claim (a process death, a swallowed dispatch failure) would block its child for good. Every enqueue — Re-run processing, Make a variant, async materialize — fails it withfailQueuedJob(conditional onqueued, so a late claimer keeps the job and the enqueue reports in progress / 409), recordschild.materialize_abandoned, and takes over. A fresh queued job still blocks.
Name reservation
A name (the key without its extension) is unique per (tenant, kind), whatever the extension. hero.png and hero.jpg cannot coexist: a variant key drops the original’s extension (m/pf/i/hero@640.webp), so two originals sharing a name would overwrite each other’s variants and deleting one would delete the other’s. SQLite enforces it with UNIQUE(tenant, kind, name); DynamoDB with a NAME# item (see media-aws); Postgres with the same unique constraint plus a per-name advisory lock (see media-postgres).
library.reserveName({ kind, filename, mime?, tenant? })holds the name (and the path it will be uploaded to) for a new asset id for 30 minutes (presigned uploads use this).insertAssetatomically claims the name or consumes the asset’s own reservation. Expired reservations read as free.reserveNamealso rejects a filename whose extension disagrees with the mime (x.pngwithimage/jpeg) as aRecipeValidationError(HTTP 400) that names the corrected filename (x.jpg);jpeg/jpecount asjpg. Unknown mimes accept any extension.ingestderives the mime from the filename extension when none is given.- Private pending prefix. A presigned upload never lands at its public key: the PUT is signed for
pending/<assetId>/<key>(pendingUploadKey,parsePendingUploadKey,PENDING_UPLOAD_PREFIXinpaths.ts;parseStoredObjectKeyaccepts both forms). The CDN cannot readpending/(a bucket-policy deny on AWS, MinIO’s anonymous policy in Docker, and the local/m/**stand-in only servesm/keys), and an S3 lifecycle rule expires it after a day.registerRemoteAsset(complete) reads the object there, checks and sanitizes it in place (stillput(..., { ifMatch })), copies the result to the public key create-only (BlobStore.copy, with the asset’s object headers), deletes the pending object (a failed delete is auditedupload.pending_cleanup_failed; the lifecycle rule expires it) and then inserts the row. An object already at the public key while this upload holds the name is no registered asset’s: a copy an earlier attempt of the samecompletemade is adopted when its SHA-256 matches, anything else is replaced, but only after re-checking, right before the delete, that the name is still this upload’s pending claim (otherwise a 409name_conflict, and the public object is not touched). A reservation lost between the copy and the insert deletes the public copy again (unless a registered asset owns the name by then).completefalls back to the public key when there is no pending object: an upload presigned before the pending prefix shipped (in flight during the deploy; sanitized in place there, with the CDN purge, as before) or a retriedcompletewhose earlier attempt already published it. A refused or discarded upload (discardPendingObject,discardUpload) deletes both keys.FsBlobStorestores pending keys like any other (<root>/pending/<id>/m/..., create-only for originals). - Abandoned presigned uploads. If a PUT succeeded but
completenever ran, the object stays underpending/(never served) until the lifecycle rule expires it; each reservation has its own pending key, so the next uploader of the name is never blocked by it. An object left at the public key by an upload from before the pending prefix still blocks the name’s create-only copy, soreserveNamedeletes a blob at the target path that has no live row or reservation and records ablob.stray_removedaudit event before handing out the new reservation. - A taken name (any extension) throws
MediaNameConflictError(aMediaConflictError) withpath,suggestedName,suggestedFilenameandsuggestedPath: the next free-2,-3, … suffix (truncated to keep the filename within 36 characters). The suggestion checks the name, not the path (hero.pngtaken →hero-2.jpgfor ahero.jpgupload). HTTP hosts callmediaErrorToHttp(err)(exported from@tgmc/media; also maps other conflicts to 409, not found to 404, validation to 400, and an image@tgmc/imagerefuses before decoding (ImageInputError, re-exported here) to 413 over the byte cap or 422 for an over-large edge or unreadable bytes) and pass the result tocreateError; the 409 body isdata.code = 'name_conflict'plus the suggestion.@tgmc/media-clientparses it back withparseNameConflictData. - Deleting an asset (
library.deleteAsset,delete-asset.ts) keeps its name claimed until its objects are gone.CatalogStore.deleteAssetCascade(id, { holdNameUntil })removes the rows and, in the same atomic step, turns the name’s claim into a pending claim of<id>#delete(deleteHolderId,DELETE_HOLDER_SUFFIX) markeddiscardingforDISCARD_HOLD_MS; the original and variant objects are deleted, thenreleaseDiscardedNamefrees the name. A same-name upload meanwhile is a 409name_conflict(with a suggestion) instead of reserving the name, removing the “stray” original and having its new object deleted by the running delete. Before the cascade the delete recordsasset.delete_started({ tenant, kind, name, paths, childIds }), so a process that dies anywhere after it (a Lambda timeout) can be finished. Every step after the cascade (deleting jobs and late children, the object deletes) runs inside the failure handling: an error is audited asasset.delete_failed(the same fields pluserror), rethrown, and the name stays held. Before each object the hold is refreshed toDISCARD_HOLD_MSfrom now (re-held if it lapsed; when another asset took the name the rest is skipped), so a slow delete never outlives it. RepeatingdeleteAssetfor the gone id resumes a delete that started or failed with noasset.deletedafter it (resumeDelete, with every key either event listed): it re-holds the name for<id>#delete(or, when another asset took the name after the hold lapsed, deletes nothing and closes it withasset.deletedskipped: 'name taken by another asset'), deletes the listed keys, frees the name and auditsasset.deleted(resumed: true). A delete refused because the asset is in use is closed byasset.delete_refusedand never resumed, and a forced delete is resumed with its usage cleanup (see Usage list and delete safeguards). WithoutholdNameUntil(the store-leveldeleteAsset, transfers) the name is freed at once. The name-discard contract (tests/contract/name-discard-contract.ts) pins the hold on SQLite, Postgres and DynamoDB. The CDN purge (hooks.afterDelete) runs last; it never fails a finished delete: an error is audited asasset.invalidation_failed({ hook: 'afterDelete', paths, error }), and repeatingdeleteAssetfor the gone id retries it (asset.invalidatedonce it succeeds). LikeafterReplaceandafterRename, every hook failure is now audited, never thrown. registerRemoteAssettakes the reservedpathinstead of a blob key and checks it parses as an original key of the asset’s kind. It inserts withinsertAsset(asset, { requireReservation: true }): the name must still be this asset’s pending reservation (expired is fine while nobody else took it). An absent, foreign ordiscardingclaim is aMediaNameConflictError(409), never a fresh claim, so acompleteracing a discard cannot register an object that is being deleted.ingestreserves inside its own insert and keeps the plain claim.- Discarding a refused upload (
upload-lifecycle.ts).library.discardUpload({ assetId, path, reason })deletes a pending upload thatcompleterefuses (over the size cap; quota later) and frees its name, auditingupload.discardedwith{ path, reason }. It only removes a pending upload: the reservation must be held byassetIdfor exactlypathwith no asset row; anything else is aMediaConflictError(409) plus anupload.discard_refusedevent, and nothing is deleted. It runs in two phases:CatalogStore.markNameDiscarding(ref, assetId)marks the claimdiscarding(still a pending holder, refused to everyholdName, the same asset’s included, and to a reservation-required insert; the hold is extended to at leastDISCARD_HOLD_MS, 10 minutes, from the mark), the object is deleted, thenreleaseDiscardedNamefrees the name. So the name stays held until the file is gone, and no new presign can reuse the key meanwhile; a crash after the mark frees the name when the extended hold lapses. A lost mark (an insert committed first) refuses without deleting. A sanitize refusal inregisterRemoteAssetuses the same helper (discardPendingObject). - Repeated
complete.library.completeRegisteredUpload({ assetId, path })returns an already-registered asset as is: no read, sanitize, size check or delete (MediaConflictErrorwhenpathis not the asset’s own,MediaNotFoundErrorfor an unknown id).completetakes this branch when the path’s holder is a registered asset, so a replay never touches the live original.
Rename
library.renameAsset(id, { filename }) (rename-asset.ts) moves an asset to a new name within its (tenant, kind). The filename is normalized like an upload’s; the extension may be left out but cannot change (400). A taken name is a MediaNameConflictError (409 name_conflict with a suggestion), an asset with an active job is a MediaConflictError (409), an unknown asset 404. Both hosts expose it as POST …/assets/:id/rename with { filename } (@tgmc/media-client renameAsset), answering the asset with its new publicUrl / previewUrl.
- Hold the new name (
holdName, 30 minutes), so no upload or other rename can take it meanwhile, and record the plan as anasset.rename_startedaudit event. - Copy the original and every stored variant to the new keys, create-only (the same object headers, with the new
Content-Dispositionfilename). The copy is server-side when the blob store has the optionalBlobStore.copy(S3CopyObject,copyFilewithCOPYFILE_EXCLonFsBlobStore), else a read and a create-only put. An object already at a new key is adopted only when its bytes are identical (an interrupted earlier run; SHA-256 streamed throughdigestSha256where the store has it); anything else is a 409. A missing original is a 409; a missing variant is skipped. - Switch the catalog in one store transaction (
CatalogStore.renameAsset): the asset’sname/pathandmeta.filename, every childpath, the new name claimed and the old one held for this asset until the cleanup releases it. The transaction refuses (409) while a job is queued or running with a fresh lease, or when the asset’s children changed since the plan (SQLite and Postgres compare the stored children; DynamoDB requires an unchangedchildRev, which every child write bumps). SQLite and Postgres do it in one SQL transaction; DynamoDB in oneTransactWriteItems(asset, children, bothNAME#items).asset.renamedis audited. - Delete the old keys and release the old name, call
hooks.afterRename({ oldPaths, newPaths })(createAwsMediaLibrarywires it to a CloudFront invalidation of the old paths whenASSETS_DISTRIBUTION_IDis set) and auditasset.rename_cleaned. A failing purge is audited asasset.invalidation_failed(paths,error) and never holds the rename open or blocks the next one. The old name is re-held for the asset first; if someone else holds it (only possible after a release), the keys are theirs and are left alone.
Child writes (variants, materialize jobs) carry the asset path they were planned for. A job whose asset moved meanwhile deletes the key it wrote (unless the old name already belongs to another asset) and redoes the variant under the new key (bounded retries; an asset that keeps moving is a retryable on-ingest failure, not “asset deleted”). A metadata edit (updateAssetMeta) writes only meta and updatedAt (one DynamoDB UpdateItem), so an edit read before a rename never restores the old name or path.
Old /m/ URLs stop working. There are no redirects: catalog media is new and admin-owned, so whoever renames updates the pages that use it. Site media (/i, /v, /d) is not in the catalog and never moves (site media).
Crash safety. Old keys are only deleted after the switch, and every step can be re-run. Re-running the same rename before the switch adopts the identical copies already written. A failure before the switch deletes every copy under the new name (the asset still holds it, so they are its own, including those of a concurrent rename to the same name that failed too; only foreign bytes a copy refused to overwrite stay), releases the new name and audits asset.rename_abandoned, unless another unfinished rename of the same asset targets the same key (it may have adopted those copies; holds are re-entrant per asset), in which case it only audits (skipped: 'another rename uses the new keys'); a store error that arrives after the switch committed (a lost response) carries on. Nothing after the switch fails the rename: a failed cleanup is audited as asset.rename_cleanup_failed, and the old name stays held for the asset. Cleanup checks every planned move before deleting anything: when a new key is missing while its old key still exists, the old keys are kept, the missing copies are written again (create-only) and the cleanup fails, so the next resume finishes it. A variant missing on both sides (skipped by the copy) does not block it. library.resumeRename(id) finishes it from the asset.rename_started event that has no asset.rename_cleaned / asset.rename_abandoned; the next rename of the asset runs it first (409 while it still fails), and GET /api/media/assets/:id and the admin’s local detail read run it on every read. It also abandons a rename that died before its switch: once its 30-minute hold on the new name has lapsed, the copies are deleted and the name released (asset.rename_abandoned, reason: 'never switched'); while the asset still holds the name the rename may still be copying, and when another asset holds it by now only the event is closed. The abandon holds the name under its own holder id (<assetId>#abandon), not the asset id, so it cannot re-enter the hold of a new rename of the same asset that took the name between its read and its hold, and that rename gets a 409 while the abandon runs. deleteAssetCascade returns the keys of the rows it deleted, read in the same transaction (DynamoDB: the asset delete is conditioned on the path it read and retried when a rename moved it, up to five attempts, then a 409), so a rename that switched while the asset was being deleted cannot leave its new keys behind. deleteAsset also removes what an unfinished rename left on either side (copies of one that never switched, old keys of one that never cleaned up) unless that name now belongs to another asset, and releases its holds (its own and a leftover <assetId>#abandon hold).
Metadata edits (PATCH …/assets/:id, validateMediaMetaEdit) accept only the editorial fields and tags: width / height / durationSec are computed, any other key (filename, facets, …) is invalid (400 invalid_meta). The admin labels an asset by its current key, not meta.filename.
Upload sanitizing
MediaLibraryDeps.sanitize (createMediaLibrary({ sanitize }), createAwsMediaLibrary({ sanitize })) turns on sanitizing of uploads (sanitize-upload.ts). It is off when unset: the CLI, transferMedia and the site import never sanitize. The sanitizer is injected, so this package imports only @tgmc/image/sanitize-core and never reaches svgo, pdf-lib or ffmpeg (tests/sanitize-boundary.spec.ts fails on a static import of @tgmc/image/sanitize or /sanitize-web, svgo, pdf-lib or ffmpeg-static).
| Host | Sanitizer | Formats |
|---|---|---|
Web server (@tgmc/media-nuxt, any mode) |
createWebUploadSanitizer() from @tgmc/image/sanitize-web (sharp + svgo) |
rasters, SVG |
Admin server (core/admin) |
createFullUploadSanitizer({ ffmpegPath: resolveFfmpegPath() }) from @tgmc/image/sanitize |
rasters, SVG, PDF, MP4 |
| MediaWorker Lambda | none (it never registers uploads) | — |
- Allowlist. With a sanitizer set, only mimes in the kinds table (
isKnownMediaMime,EXTENSION_BY_MIMEinkinds.ts) are accepted, at presign,ingestandcomplete; anything else (text/html,application/octet-stream, HEIC, …) is a 415unsupported_type(MediaUnsupportedTypeError,data: { code, mime }). Content is also sniffed for markup (sniffMarkupfrom@tgmc/image/sanitize-core): HTML is always refused, XML unless it is a declared SVG, and any markup in a type with no sanitizer format (a.txt, an audio file). For a stored upload the check reads the first 1024 bytes through the optionalBlobStore.getPrefix(S3 rangedGetObject, a partial read onFsBlobStore), else the whole object when it is within the upload cap. ingestsanitizes the bytes before hashing and storing them, and auditsasset.sanitizedwith the sanitizer’s report.- SVG never uses a presigned upload.
assertUploadSanitizable({ filename, mime, preSanitized })refuses every SVG at presign (415sanitize_required, pointing at the body route), even withpreSanitized: true: the object would sit unchecked untilcomplete(under the privatepending/prefix since it exists, but an SVG is active content and is only ever accepted sanitized, in the request body). The admin sends the sanitized SVG in the request body instead (POST /api/media/ingest/svgin media-nuxt,ingest({ kind: 'svg', preSanitized: true })), and the web server sanitizes it again before anything is stored. - SVG is always run through svgo when the sanitizer supports SVG (web and admin),
preSanitizedor not, and the result is then scanned (findSvgActiveContent): what svgo leaves active is a 415invalid_upload, and so is markup that is not plain UTF-8 (invalid UTF-8, another declared encoding, control characters), before any tool runs. Only the sanitized bytes are stored, and only when they still have an<svg>element: svgo turns a comment-only, empty or XHTML file into empty output, and that is a 415invalid_uploadtoo (never an empty asset). A sanitizer without SVG support (createRasterUploadSanitizer(), still re-exported from@tgmc/mediafor hosts that want rasters only) keeps a pre-sanitized SVG only when the scan finds nothing (else 415sanitize_required) and it has an<svg>element. - Presigned uploads (presign → PUT → complete). In
presign, before the name is reserved, a PDF or MP4 is a 415sanitize_requiredunless the request sayspreSanitized: true(the admin server ran the full sanitizer before forwarding). The PUT goes to the private pending key (Name reservation). Incomplete,registerRemoteAsset({ preSanitized })reads the uploaded object there (headfor its ETag, thenget), sanitizes it and writes the result back in place withput(key, bytes, { ifMatch: etag }), then copies it to its public key, before the row is inserted. The conditional put (S3If-Match, a check-then-rename onFsBlobStore) refuses to overwrite an object that changed since the read (MediaBlobChangedError, whichcompletereports as a 409 conflict; complete it again). Thenasset.sanitizedis audited (inPlace: false: the public key only ever held the sanitized bytes). Only an upload from before the pending prefix is sanitized at its public key (inPlace: true), and only then doesafterReplacepurge the CDN; a hook failure is audited asasset.cdn_invalidate_failedand does not fail the upload. - Refusals. A file whose content is another sanitizer format than its name says is 415
sanitize_required. A PDF or MP4 whose bytes are not one (refused before any tool runs), or a file the sanitizer cannot read, is 415invalid_upload(MediaUploadInvalidError). A raster or SVG overMAX_BYTES(25 MiB) and a PDF to be sanitized overMAX_PDF_SANITIZE_BYTES(10 MiB, see image § sanitizers) are a 413 before anything is read. Every PDF, pre-sanitized or not, is also capped atMAX_PDF_SANITIZE_BYTESwhen it is presigned: thedocumentupload cap (uploadCapBytes('document'), see media-nuxt § Upload size cap) is the same 10 MiB, and the admin’s own sanitizer refuses larger PDFs anyway. A refused presigned upload’s object (sanitize_required,unsupported_type,invalid_upload, or over the size cap) is deleted and its name released, in two phases (see Name reservation). Only a missing tool (an error withcode: 'FFMPEG_NOT_FOUND') is a 503 (MediaSanitizeUnavailableError). - Residual risks. Between the presigned PUT and
complete, an unsanitized raster, PDF or MP4 sits under the privatepending/prefix, which CloudFront cannot read (closed by the pending prefix; before it, the object sat at its public key). CloudFront adds noContent-Security-Policy: sandboxto/m/*yet.preSanitizedis trusted from any holder of the admin token (the admin’s HMAC, media-nuxt § AdminpreSanitizedsignature, already refuses a wrong signature; an unsigned one is refused once Phase 7 S1-13 enforces it). MP4 and PDF uploads straight to the web API need the admin path. All three are follow-ups in the TODO.
S3 object headers
BlobStore.put(key, data, { contentType, cacheControl, contentDisposition, createOnly }). createOnly refuses to replace an existing object: it throws MediaBlobExistsError, sent as If-None-Match: * on S3 (where 412 and 409 map to that error) and published through an exclusive link on FsBlobStore. MediaLibrary passes the asset mime, public, max-age=31536000, immutable and inline; filename="<name>.<ext>" (objectHeaders(contentType, key), also used by transferMedia). S3BlobStore sets them as object metadata; FsBlobStore keeps them in its sidecar (below). Presigned browser uploads sign the full PRESIGN_REQUIRED_HEADERS set (Content-Type, If-None-Match, Cache-Control, Content-Disposition; see media-aws), so those objects carry the same cache and download headers as server-side puts.
Versioned public URLs
The one-year immutable cache is kept, but a key can get new bytes: a catalog variant is re-materialized in place, an upload is re-encoded at complete, and a name is reused after a delete or rename. So every public media URL carries the object’s content version, ?v=<first 8 hex of its SHA-256>, built in one place: mediaPublicObjectUrl(base, key, version) in src/public-url.ts (browser-safe, no imports; also exported as @tgmc/media/public-url). The version of an original is its sha256 (mediaAssetObject); of a variant, its meta.sha256, which runMaterializeJob records for every ready child (mediaChildObject); a variant materialized before that falls back to an 8-hex digest of its updatedAt (mediaObjectVersion), which every re-materialize bumps. mediaPreviewObject(asset, children) is mediaPreviewKey plus the previewed object’s version. New bytes get a new URL, so neither CloudFront (whose m/* cache key includes v, see infra) nor a browser serves a stale copy; the stores ignore the query (S3, MinIO and the local /m/** stand-in serve the key). The CDN purges (afterReplace, afterDelete, afterRename) still run and invalidate every cached version of the key (/key*), including links that kept an old v. mediaAssetUrls(base, asset, children) gives both an asset’s publicUrl and previewUrl, and withMediaAssetUrlsList(library, assets, base) does it for a list page (only image / svg rows read their children). Users: @tgmc/media-nuxt (publicUrl / previewUrl, Asset URLs), the admin’s local mode (both through those two helpers), and sitePublicMediaUrl(key, runtimeConfig, sha256) in core/web/shared/media-public.ts (import-free copy, pinned to the same result by tests/media-public.spec.ts).
Filesystem blob store
FsBlobStore(root) is the local stand-in for the S3 bucket, so host dev and DATA_BACKEND=sqlite|postgres behave like AWS:
- Layout. A blob lives at
<root>/<key>(<root>/m/pf/i/manhattan-keyframe.png): the disk path, the S3 key and the URL are the same string.createMediaLibraryuses<blobDir>;@tgmc/media-nuxtuses<MEDIA_LOCAL_ROOT>/blobs. - Object metadata. A
<key>.meta.jsonsidecar holdscontentType,byteSize,sha256,etag(the quoted MD5, as S3 returns for a single PUT) and, when given,cacheControl/contentDisposition(FsBlobMeta,FS_BLOB_META_SUFFIX).readMeta(key),head(key)(same shape asS3BlobStore.head) andstat(key)read it. The sidecar also records ageneration(inode, mtime in ns and size of the file it was written for). A variant’s body and sidecar are published by two renames, so a reader between them, or a crash, could otherwise pair new bytes with the old ETag/SHA-256. A blob without a sidecar, or whose sidecar was written for another file (a same-size variant replacement, blobs moved by the SQLite migration below, files copied in by hand), gets metadata derived from the file: type from the extension, hashes computed on read. - Create-only originals. A write to an original key (no
@variant) goes to a temp file and is published with an exclusive hard link (an exclusive copy where links are unsupported, and always on Windows: there a removed temp link can stay delete-pending while another process such as a scanner holds the just-written file, and until it clears the object cannot be replaced, so the in-placeifMatchsanitize re-encode failed with EPERM;createOnlyPublishMethod()). A second write to the same original throwsMediaBlobExistsError, whichmediaErrorToHttpmaps to 412, like a presigned PUT withIf-None-Match: *. A publish by copy that fails part-way (disk full, I/O error) removes the truncated target it created (COPYFILE_EXCLproves it was this call’s), so the key is not left blocking the next create-only write; a/m/**reader could see the partial bytes only for the instant before the removal. Variant keys overwrite in place (temp file + rename), as catalog re-materialize needs, unless the put passescreateOnly(the transfer does).deleteremoves the blob and its sidecar, so the name can be uploaded again. - Streaming.
putStream(key, source, { maxBytes, ...headers })hashes while writing and never buffers the whole body; pastmaxBytesit throwsMediaBlobTooLargeError(413) and leaves nothing behind.digestSha256(key)re-reads the file.openRead(key)serves it: it opens the file once and returns{ meta, stream(), close() }, wheremetadescribes exactly the bytesstream()reads, even if the key is overwritten mid-request. - Key guard. Every operation first checks that the key is a valid media key (
parseMediaKey) with no empty,.or..segment, no backslash and no NUL, and that it resolves inside the root. Anything else (traversal, absolute or drive paths, sidecars, temp files) is aRecipeValidationError(400) before the disk is touched, so sidecars and temp files can never be read through a key.
SQLite migration
openMediaDatabase(dbPath, { blobDir }) migrates older databases in place and idempotently: it adds tenant, name, path (assets) and variant, path (children), backfills names from meta.filename (else {kind}-{id8}, de-duplicated with numeric suffixes), marks old children as custom, renames later assets that collide on (tenant, kind, name) across extensions with -2, -3 (their variants are re-keyed; when two assets shared a variant key, the one object at that key cannot be attributed, so both children are marked failed and need re-materializing), creates the unique (tenant, kind, name) index (replacing the old path index and media_path_reservations table with media_name_reservations; a colliding row whose key cannot be parsed aborts the migration with an error naming the row), then moves blobs on disk from the legacy blob_key to the new path. blob_key stays as a legacy column. A missing file keeps its row pending for the next run; an existing destination marks the row done. Moved blobs have no sidecar; FsBlobStore derives their metadata. Take a copy of data/ first.
Kind vs tags / meta
| Field | Role |
|---|---|
kind |
Exclusive processing pipeline (image / svg / audio / video / model / document) |
meta.tags |
Many editorial labels (hero, lottie, project names) — normalized lowercase kebab |
meta facets |
Pipeline hints without new kinds: animated, hdr, panorama, designSource, motionFormat (ASSET_META_KEYS) |
List with ?tag= (admin + /api/media). PATCH …/assets/:id with { tags } or { meta } to update (a validated edit, see Metadata policy).
Usage list and delete safeguards
A usage record says that a published page uses an asset: MediaUsageRecord is { tenant, pageType, pageId, assetId, pageLabel, url, state, updatedAt }. state is live (the page is published with the asset) or pending (written ahead of a publish that may not land; a rebuild removes it once it is older than PENDING_USAGE_GRACE_MS, 15 minutes). A page is one of four MediaPageTypes (MEDIA_PAGE_TYPES): post, post-cover, slot, site. Drafts never count.
The UsageStore port (ports/index.ts; adapters implement it next to their catalog store):
replacePageUsage(page, { assetIds, pageLabel, url, state, updatedAt }): make the page’s rows equalassetIds, atomic per asset and idempotent; returns sorted{ added, removed, missing }(missing: ids with no asset in the page’s tenant, which get no row).listUsageForAsset(assetId): rows ordered bypageType, thenpageId(byte order).listUsageForPage(page): rows ordered byassetId.removeAssetUsage(assetId): remove and return every row of the asset.rebuildUsage({ tenant, pageTypes, pages, now }): within that tenant and those page types only, make the rows equalpages(alllive); never removes or shrinks a row younger thanPENDING_USAGE_GRACE_MS, whatever its state. Every adapter plans the rebuild with the storage-freeplanUsageRebuild(existing, listedUsageRows(input, now), cutoff)(usage-rebuild.ts;usageRebuildTimes(now)gives the cutoff) and only runs itsremove/rewrite/recheck/insertsteps, so the grace rule and the “changed” test live in one place. A catalog adapter checks a rename the same way withassertNoUnplannedChild(input, storedIds).
DeleteAssetCascadeOptions.usage is 'refuse' (default) or 'ignore'. With 'refuse' the cascade reads the asset’s usage rows inside its atomic step and, if any exist, throws MediaInUseError(assetId, pages) (pages ordered like listUsageForAsset, pending rows included) and changes nothing. With 'ignore' it deletes the asset and leaves the rows for removeAssetUsage. parseForceQuery(value) is the shared parser for a ?force= query (undefined/'0' is false, '1' is true, anything else 'invalid').
mediaErrorToHttp sends MediaInUseError as 409 in_use (checked before the generic conflict branch):
{
"statusCode": 409,
"statusMessage": "Asset a-1 is used by 1 published page",
"data": {
"code": "in_use",
"pages": [
{ "pageType": "post", "pageId": "p-1", "pageLabel": "Harbor notes", "url": "/blog/harbor-notes", "state": "live" }
]
}
}
Library methods. MediaLibrary takes the store as deps.usage (createMediaLibrary passes the SQLite one); each usage method throws Error('MediaLibrary has no usage store') without it.
getAssetUsage(id): the asset’s pages (MediaUsagePage[],listUsageForAssetorder);MediaNotFoundErrorwhen the asset is gone.replacePageUsage(page, { assetIds, pageLabel, url, state }): dedupesassetIds, stampsupdatedAtwith the current time, and throwsRangeErrorfor a page type outsideMEDIA_PAGE_TYPESor a blankpageId.rebuildUsage({ tenant, pageTypes, pages }): validates the same way (every listed page type, every page’s type and id) and stampsnow. This is the hook the admin’s rebuild endpoint and the post-deploy run call.
Delete sequence (deleteAsset(id, { force? }), delete-asset.ts):
- The running-job check (409 while a materialize job runs).
- Pre-check: without
force, the usage store’slistUsageForAsset; any row (liveorpending) is aMediaInUseErrorwith the pages, and nothing is deleted or audited. asset.delete_startedis recorded, then the atomic check: the cascade runs withusage: 'refuse'and reads the rows again inside its atomic step, so a page published between the pre-check and the cascade still refuses it. That refusal changes nothing, is audited asasset.delete_refused({ pages }) and rethrown. The event closes the delete:resumeDeletetreats it likeasset.deletedand never finishes a refused delete, even if the asset’s rows disappear some other way later.- From here the delete runs as before (name hold, objects,
asset.deleted).
Force ({ force: true }, the admin’s override): it needs the usage store (else it is refused before anything changes). It reads the pages and records asset.delete_started with force: true and those pages, then runs the cascade with usage: 'ignore'. With 'ignore' the cascade returns the usage rows it read inside its atomic step (DeleteAssetCascadeResult.usage, on the returned key array). After the cascade no usage write adds a row for the gone asset (it reports the id in missing), but a page write or a rebuild can still remove one. So the delete audits asset.force_deleted ({ tenant, pages }, pages ordered by pageType, then pageId) with the union of the rows the cascade read and the rows listed again now. A page published between the pre-read and the cascade is named too. Only then does it remove the rows (removeAssetUsage) and finish as above. The audit comes before the remove, so a crash in between loses no page from the record. A failure after the cascade (including either of those steps) is asset.delete_failed like any other step. Repeating deleteAsset for the gone id (with or without force; the audit trail says whether it was forced) resumes it. For an open asset.delete_started with force: true, resumeDelete appends asset.force_deleted only when none follows it (with the union of delete_started.pages and the rows listed now), and always calls removeAssetUsage (idempotent), before finishing the objects (asset.deleted, resumed: true). The resume never re-runs the cascade, so the rows a forced delete leaves behind (which make a plain cascade of the gone id refuse) cannot stall it. The one residual gap, for an adapter that returns .usage (every adapter passing the usage contract does): only after a crash between the cascade and the asset.force_deleted audit can a page go unnamed, and only one that was both published after the pre-read and then republished or rebuilt in that window. If recording asset.delete_refused itself fails, the caller still gets the MediaInUseError (409), not the audit error.
The admin offers a forced delete only from a refusal (plan 4, admin § Used by, refusal and Delete anyway): Delete anyway appears only after Delete… found pages, so a forced delete from the admin always has pages and asset.force_deleted names them. The library itself still accepts force on an unused asset (it audits force_deleted with pages: [] and returns forced: true); only a direct API caller can do that.
The admin server passes all of this through in both its modes (deleteMediaAsset(event, id, { force }), getMediaAssetUsage, rename usedBy; admin § Admin API), and @tgmc/media-client turns a 409 in_use into MediaInUseError (media-client).
A usage write for an asset whose delete has not finished finds no asset row and reports the id in missing. Rename never reads or changes usage: rows key on the asset id, so a renamed asset keeps its pages. The library delete itself is immediate; the 8-second Undo is the admin’s client-side delay before it sends the delete, unchanged.
SQLite (SqliteUsageStore, in createSqliteMediaStores as usage and passed to MediaLibrary by createMediaLibrary): the media_usage table in the catalog database, keyed by (asset_id, tenant, page_type, page_id), with an index on (tenant, page_type, page_id) for page reads. SCHEMA creates it with CREATE TABLE IF NOT EXISTS, so existing databases gain it on their next open. Each method is one transaction. A row is written only while the asset exists in the page’s tenant (INSERT … SELECT … WHERE EXISTS on media_assets); an id that finds no asset lands in missing and a row left behind by a forced delete is dropped the same way. SqliteCatalogStore.deleteAssetCascade reads the rows in its own transaction before the first delete, so a usage write and a delete never both succeed. rebuildUsage runs as one transaction: rows of the listed types that pages does not list are deleted, listed rows are inserted or rewritten as live (an old listed row whose asset is gone is deleted and counted in removed), and a row written less than PENDING_USAGE_GRACE_MS before now is neither deleted nor rewritten. countersRepaired is always 0 (SQL keeps no counters).
Postgres (PostgresUsageStore, @tgmc/media-postgres): the same table and rules, with row locks instead of SQLite’s single writer: a per-page advisory lock, FOR KEY SHARE on each asset a write adds or keeps (against the cascade’s FOR UPDATE), and the cascade reading the rows after its row lock. See media-postgres § Concurrency.
DynamoDB (DynamoUsageStore, @tgmc/media-aws): rows under the asset, a mirror per page, a per-asset counter that changes in the same transaction as every row, and a page head whose revision every chunk checks and bumps. The cascade’s ConditionCheck on the counter (index 2 of its transaction) is the atomic check; a forced cascade’s check is “counter unchanged since read”, so its .usage rows are the rows at the delete. A counter above zero with no rows is refused as a conflict naming the rebuild, and rebuildUsage repairs counters (countersRepaired). Writes go in chunks of 20 assets (the 100-item transaction limit): each chunk commits only against the state it planned from; writers of a multi-chunk page may alternate chunks, and the last to finish wins. See media-aws § Keys and DynamoDB layout.
Times, every adapter. Each usage store passes updatedAt (replacePageUsage) and now (rebuildUsage) through normalizeUsageTime(value, field) before its transaction: stored as UTC ISO with milliseconds (rows are compared and ordered as strings), and a value that is not a time throws a plain Error naming the field, before anything changes.
Catalog contract tests
tests/contract/catalog-contract.ts is one suite for every CatalogStore / JobStore / AuditStore adapter: describeCatalogContract(label, harness) with a harness that opens fresh stores (open()), releases them (close(stores)) and, optionally, shared resources (dispose()); an undefined harness skips the suite. It pins the SQLite adapter’s behaviour: round-trips (numbers, nested meta, timestamps), list order and paging, tenant / kind / tag filters, conflict errors (MEDIA_CONFLICT, MediaNameConflictError across extensions), name holds (extend, consume, expiry, release, one winner under concurrency), the two-phase discard (markNameDiscarding / releaseDiscardedName and the reservation-required insert; tests/contract/name-discard-contract.ts, which the DynamoDB adapter also runs through describeNameDiscardContract), child writes guarded by their asset, finalize outcomes, job transitions (one concurrent claim wins) and audit order. Errors are matched by name / code, because an adapter package throws the classes of its own built @tgmc/media.
- SQLite:
tests/sqlite-contract.spec.ts(in-memory database), always runs. - Postgres:
packages/media-postgres/tests/postgres-contract.spec.ts, whenMEDIA_PG_TEST_URLis set (media-postgres § Tests).
A new adapter adds one spec with its harness.
tests/contract/usage-contract.ts does the same for UsageStore and the usage refusal of deleteAssetCascade: describeUsageContract(label, harness), where open() returns { catalog, usage } over the same data with no usage rows at all and no assets with the case’s ids: a fresh database / schema / table per case, or the shared one truncated before each case (rebuildUsage works on a whole tenant and page type, so rows left by an earlier case would be removed and counted). It covers row writes and reads, the difference a replace writes, missing and other-tenant assets, the refusal (live and pending rows, page order, nothing changed), usage: 'ignore' plus removeAssetUsage, time normalization and invalid times, and rebuildUsage (listed types and tenant only, old pending rows removed, old rows of gone assets dropped, rows younger than the grace period never removed or rewritten). Its race case — a usage write racing deleteAssetCascade either refuses the delete or records the asset missing, never both — must pass both branches on a backend with concurrent writers; rounds alternate which call starts first, and the test never asserts which one won. SQLite (tests/sqlite-contract.spec.ts) runs it always; with one synchronous writer the start order decides the branch there. Postgres (packages/media-postgres/tests/postgres-contract.spec.ts, when MEDIA_PG_TEST_URL is set) runs it unchanged, with a fresh schema per case; there the cascade usually takes the row lock first whatever the start order, so that package adds lock tests that hold each side open to force both orders (media-postgres § Tests). DynamoDB (@tgmc/media-aws) runs it unchanged on the in-memory DocumentClient (tests/dynamo-usage.spec.ts) and, when DYNAMODB_TEST_ENDPOINT is set, on DynamoDB Local with a fresh table per case (tests/dynamo-local.contract.spec.ts); there too the cascade commits first when left alone, so that file adds two cases that hold each side at its transaction (media-aws § Keys and DynamoDB layout).
Transfer between backends
transferMedia copies a catalog and its objects from one backend to another through the ports only (no AWS import in this package). It seeds production from a local catalog and pulls backups the other way. Operators use the npm run media:transfer CLI (runbook).
import { createSqliteMediaStores, transferMedia, type MediaEndpoint } from '@tgmc/media';
import { S3BlobStore, createDynamoMediaStores } from '@tgmc/media-aws';
const local = createSqliteMediaStores({
dbPath: './data/admin-media/media.sqlite',
blobDir: './data/admin-media/blobs',
});
const dynamo = createDynamoMediaStores({ tableName: 'portfolio-prod-Media', region: 'us-west-2' });
const target: MediaEndpoint = {
catalog: dynamo.catalog,
audit: dynamo.audit,
blobs: new S3BlobStore({ bucket: 'portfolio-prod-assets-<account>', region: 'us-west-2' }),
};
const report = await transferMedia({ source: local, target, dryRun: true });
local.close();
- Endpoints.
MediaEndpoint = { catalog, blobs, audit? }.createSqliteMediaStores({ dbPath, blobDir })returns the SQLite stores plus anFsBlobStoreandclose()(createMediaLibraryis built on it). WithreadOnly: truethe catalog is never created, migrated or written: an existing file opens as it is, a missing one reads as an empty catalog, and catalog writes throw (combining it with a pre-openeddatabasethrows). The CLI uses this for a dry-runsqlite:target. A database created before the usage list and opened read-only has nomedia_usagetable (only a writable open adds it), so a usage read on it throwsno such table; no read-only caller lists usage today, and one that does must open the file writable once first or treat that error as “no usage”.transferMediacopies catalog rows and objects, not usage rows: run the usage rebuild on the target after a transfer. On AWS, combinecreateDynamoMediaStoreswithS3BlobStore(media-aws). On Postgres, combinecreatePostgresMediaStores(readOnlyfor a dry run) withS3BlobStoreorFsBlobStore(media-postgres). - Options.
dryRun(read both sides, write nothing),includeChildren(defaulttrue),pageSize(default 100),tenants(default: every tenant),correlationId(recorded on the audit events; default: a new id per run),auditAction(defaultasset.transferred). The source is read page by page withlistAssetsfor each tenant. The source only needsMediaTransferSource(catalog.listAssets/listChildren,blobs.get/digestSha256), so an in-memory source works (the site import uses one). -
One tenant (CLI).
npm run media:transferpasses--tenant <name>through astenants. It is repeatable (--tenant portfolio --tenant example-organization; kept in order, duplicates dropped), and without it every tenant is copied. The names are those ofTENANT_CODES(portfolio,example-organization); any other value is a usage error (exit2). The CLI keeps its own copy of the list (MEDIA_TENANT_NAMESinscripts/media-transfer-cli.js, which imports no workspace package), andscripts/media-transfer.test.jsfails if it drifts frompaths.ts. A per-tenant backup, for example Example Organization’s:npm run media:transfer -- --from aws:<table>@<bucket> --to sqlite:./backups/ha --tenant example-organization - Copy. Each object is read from the source and checked against the catalog
sha256. It is written with the same headersMediaLibraryuses (objectHeaders: the mime,public, max-age=31536000, immutable,inline; filename="<name>.<ext>"), then re-hashed on the target (BlobStore.digestSha256, optional on the port; implemented byFsBlobStoreandS3BlobStore). Only after that check passes is the row written withinsertAsset. The row keepsid,tenant,name,path,metaand the timestamps, so URLs are the same in both environments. On DynamoDB,insertAssetwrites theNAME#claim in the same transaction. IfinsertAssetthrows but a re-read finds the row with the samesha256(a timeout after the commit), the asset counts as copied and its object stays. If the re-read itself throws, the object also stays (it may back a landed row; an orphan object is harmless and the next run adopts it) and the asset is recorded infailed. Child inserts follow the same rule, re-reading withlistChildren. Before adopting an object already at the key, the source object is still hashed and checked against the catalogsha256. On DynamoDB,getAssetis a strongly consistent read, so the re-read and the skip check see a row that just committed. - Concurrent uploads on the target. Before writing anything (not in a dry run), the transfer holds the asset’s name on the target with
holdNameforTRANSFER_NAME_HOLD_MS(15 minutes). This is the same reservation an admin upload takes. If the hold is refused, the asset is a conflict and nothing is written. On any failure beforeinsertAssetsucceeds,releaseNamefrees the hold;insertAssetconsumes it on success (SQLite deletes the reservation in its transaction, and DynamoDB’sNAME#claim replaces it). Objects are written withBlobPutOptions.createOnly(S3If-None-Match: *;FsBlobStorepublishes through an exclusive link). An object that appears at the key between the existence check and the put (412, or 409ConditionalRequestConflict, surfaced asMediaBlobExistsError) is therefore a conflict. It is never overwritten, and the cleanup never deletes it. Remaining gap: children are not covered by the name hold. A media worker on the target re-materializing the same variant key overwrites in place (variants are not create-only for workers). So if it writes between the transfer’s create-only put and a failed child insert, the transfer’s cleanup can delete the worker’s object. The worker’s child then needs to be materialized again. Keep the target quiet during a transfer. - Children. Only
readychildren with a path are copied, keeping their ids and variant keys (m/pf/i/hero@t.webp), throughinsertChildIfAssetExists. A child whose path is not a variant of its asset’s name fails. - Reruns are idempotent. If the target already has the same id with the same
sha256and path, and its object exists, the asset isskipped(the object is not re-hashed; a child with the same id and path is skipped on the same terms). Any ready children it lacks are still copied, so a rerun finishes a half-done copy. An identical object already at the key, with no row (an earlier run stopped after the write), is reused rather than written again. - Conflicts are never overwritten. These are reported in
conflicts, with the reason inconflictReasons[id]:- the same id with another
sha256or path - the name (tenant, kind, name, any extension) held on the target by another id, including a pending upload, whether it was held before the run or taken during it
- different bytes already at the key, or an object that appeared there during the run (
MediaBlobExistsErrormaps to a conflict) - a child id or path already used on the target
- the same id with another
- Failures. These go into
failed({ id, error }) and the run continues: a missing or corrupt source object, a target object that fails the re-hash, a failed insert, or an asset or child already on the target whose object is missing there. The run never writes a row without a verified object, and it normally deletes any object it wrote for a failed item. The exceptions keep data in place: a failed insert whose re-read also failed keeps the verified object (its row may have landed; a rerun adopts it), an existing row whose object is missing stays as it is, and anaudit: …failure keeps everything that was copied. A sourcelistAssetserror stops that tenant’s listing and is recorded aslisting <tenant> (offset <n>). The other tenants still run, and the partial report is returned. - History. Jobs and audit history are not copied. For every asset the run wrote (the asset or any of its children), the target gets one
asset.transferredaudit event (TRANSFER_AUDIT_ACTION). Its payload is{ path, sha256, assetCopied, children }. If appending the event fails, the copied data stays and the failure is added tofailedasaudit: …. - Report.
TransferReporthascopied/skipped/conflicts/conflictReasons/failedfor asset ids, the same fields underchildrenfor child ids,dryRun, andsourceCount(source assets seen). In a dry run,copiedlists what would be copied. A dry run still reads and hashes the source objects, so it also catches missing or corrupt ones. - Don’t write to the source while a transfer runs. Paging is by offset (SQLite orders by
created_at DESC, id DESC), so new rows could shift the pages. Opening a SQLite catalog also runs its migrations (SQLite migration).
Site media manifest and import
The site’s own media is described by a reviewed manifest (core/web/content/media.json) and imported with fixed ids. Workflow and CLIs: site media.
site-manifest.ts:SiteMediaEntry/SiteMediaManifesttypes,validateSiteManifest(value)→{ manifest | null, errors, metaIssues }(strict keys, path / kind / name / ext agreement, anextwhose MIME type belongs tokind, UUID ids, sha256, catalog variants,legacyPathwithout./..segments, uniqueness of id, name and legacy path; policy issues per id),siteMediaId(legacyPath)(UUIDv5 underSITE_MEDIA_ID_NAMESPACE),uniqueSiteMediaName,siteEntryToAsset/siteEntryFromAsset(round trip with the catalog; meta getstags: ['site'], neverreviewNotes),serializeSiteManifest(stable key order, trailing newline).site-import.ts:importSiteMedia({ items: [{ entry, bytes }], target, dryRun?, allowUnreviewed?, correlationId? })(bytesis aBufferor a loader, called only for entries it imports). Refuses entries with review notes or policy issues, skips entries the target already has with the same id, path andentry.sha256(checked before the bytes are loaded, so reruns survive sanitizer upgrades and need no ffmpeg), fails entries whose loader throws or whose bytes do not matchentry.sha256, and copies the rest throughtransferMedia(in-memory source, no children,auditAction: 'asset.site_imported').siteImportSucceeded(report)is false on any refusal, conflict or failure.
CLI
npm run build --workspace=@tgmc/media
node packages/media/bin/media.mjs smoke --db /tmp/m.sqlite --blob-dir /tmp/m-blobs
bin/media.mjs is the only executable entry (it owns process.exit). src/cli.ts exports runCli(argv) → exit code and must stay free of import-time side effects: it is bundled into server builds via the package entry, where a “run if main” guard fired and exited the admin server (solution; guarded by tests/cli-import.spec.ts).
Boundaries
- Node-safe root: the package root also exports the fs / SQLite / crypto adapters, so client code must not import it.
@tgmc/media/kindsis the one browser-safe entry (src/kinds.ts, no imports):MEDIA_KINDS,isMediaKind,guessMediaKind,defaultMimeForKind,extensionForMime,extensionMatchesMime,mimeForExtension. The admin media page uses it (core/admin/tests/media-page-ui.spec.tsfails if a client file imports the root).- Does not replace
/api/image/*on-demand transforms; those stay on@tgmc/image+ web handlers. - Does not own CDN upload or CloudFront invalidation: it calls the
afterReplace,afterDeleteandafterRenamehooks, which@tgmc/media-awswires to CloudFront.