@tgmc/media-aws
AWS adapters and SQS worker for @tgmc/media.
Package README: packages/media-aws/README.md.
Role
| Concern | Owner |
|---|---|
| DynamoDB catalog / jobs / audit | This package |
S3 blobs under m/ on AssetsBucket |
This package |
| Presigned PUT ingest (reserved key) | createPresignedIngest, PRESIGN_REQUIRED_HEADERS |
| SQS enqueue + worker (materialize jobs, S3 upload events, re-runs) | This package (upload triggers) |
| CloudFront invalidation on delete/replace | invalidateMediaPaths / MediaLibrary hooks |
| HTTP routes | @tgmc/media-nuxt /api/media/* (Bearer ADMIN_TOKEN) |
| IaC | infra/sam Media table, queue, worker, Nitro IAM |
Requirements: docs/plans/2026-09-22-002-feat-media-aws-backend-plan.md.
HTTP API (Bearer admin)
| Method | Path | Notes |
|---|---|---|
| POST | /api/media/ingest/presign |
{ kind, mime, filename, tenant?, preSanitized? } → reserves m/{tenant}/{kind}/{name}.{ext}, returns { assetId, path, name, expiresAt, uploadUrl, headers }; 409 name_conflict with a suggested name; 415 sanitize_required for every SVG (body route) and for PDF / MP4 unless preSanitized; 415 unsupported_type outside the kinds allowlist |
| POST | /api/media/ingest/complete |
{ assetId, path, kind, mime, filename?, meta, sha256?, preSanitized?, preSanitizedSig? } (an admin PDF / MP4’s flag is checked against the HMAC, see media-nuxt); path must be the key reserved for assetId (400 otherwise); sanitizes the object in place (If-Match) before the row is written |
| PATCH | /api/media/assets/:id |
{ meta?, tags? }, validated (400 invalid_meta) |
| POST | /api/media/assets/:id/rename |
{ filename } → new keys for the original and variants; old paths invalidated on CloudFront (afterRename) |
| GET | /api/media/assets |
list (kind, tenant, tag, limit, offset) |
| GET | /api/media/assets/:id |
asset + children + processing (on-ingest step status) |
| POST | /api/media/assets/:id/process |
Re-run processing: queues { type: 'on-ingest', assetId } for the worker (enqueueOnIngestRecipes); idempotent |
| POST | /api/media/assets/:id/materialize |
{ role?, variant?, format?, recipe?, useDefaultThumb? } → enqueue (or sync locally); default is the t thumbnail |
| GET | /api/media/jobs/:id |
job status |
| DELETE | /api/media/assets/:id |
hard cascade + invalidation when configured (a failed purge is audited asset.invalidation_failed, not a 500; DELETE again retries it) |
Keys and DynamoDB layout
Object keys are the catalog path (m/{tenant}/{kind}/{name}.{ext}, variants …@{variant}.{ext}); toS3Key is the identity for m/ keys and toCloudFrontPath is / + key. See media for the scheme.
| Item | Key |
|---|---|
| Asset | pk = ASSET#{id}, sk = META; gsi1pk = T#pf#K#i (tenant + kind code), gsi2pk = T#pf#ALL |
| Child | pk = ASSET#{id}, sk = CHILD#{childId} (+ CHILDLOOKUP#{childId}) |
| Name / reservation | pk = NAME#m/{tenant}/{kind}/{name}, sk = META → { assetId, path, pending, expiresAt?, ttl? } |
| Usage row | pk = USE#{assetId}, sk = PAGE#{pageType}#{pageId} → the MediaUsageRecord fields (no GSI) |
| Page mirror | pk = PAGEUSE#{tenant code}#{pageType}#{pageId}, sk = USE#{assetId}; gsi2pk = T#pf#USAGE, gsi2sk = {pageType}#{pageId}#{assetId} |
| Usage counter | pk = USE#{assetId}, sk = COUNT → { n, rev, tenant }; gsi2pk = T#pf#USAGECOUNT, gsi2sk = {assetId} |
| Page head | pk = PAGEUSE#{tenant code}#{pageType}#{pageId}, sk = HEAD → { rev } (no GSI) |
The NAME# key has no extension, so hero.png and hero.jpg share one item and the second is a 409 name_conflict with a -2 suggestion (the suggestion search reads NAME# items, up to 1000 consistent reads in the worst case). insertAsset writes the asset and its NAME# item in one transaction (attribute_not_exists on both; the name item may also be the asset’s own pending reservation or another asset’s expired one). A pending reservation carries expiresAt and a numeric epoch-seconds ttl; the completed claim is permanent (pending = false) and carries neither.
DynamoDB TTL. Enable TTL on attribute ttl for the media table so lapsed reservations are deleted (infra/sam/template.yaml sets TimeToLiveSpecification: { AttributeName: ttl, Enabled: true } on MediaTable; docker/dynamodb-init.sh enables it on the docker:test Media table too). TTL deletion is lazy (up to ~48 h), so reads still compare expiresAt and treat an expired reservation as free. The store’s deleteAsset removes both in one transaction, so a deleted name is reusable at once; deleteAssetCascade(id, { holdNameUntil }) (what MediaLibrary.deleteAsset calls) instead replaces the NAME# item, in the same transaction, with a pending discarding claim of <id>#delete (with ttl) until the objects are deleted (media § Name reservation). listAssets queries the tenant-scoped GSI partitions.
Usage (DynamoUsageStore, src/dynamo-usage.ts; createDynamoMediaStores returns it as usage and createAwsMediaLibrary passes it to MediaLibrary). It lives in MediaTable with the items above: no new table, no IAM change. Fields are stored as attributes and the keys are never parsed (keys.ts: usagePk, usageSk, USAGE_COUNT_SK, pageUsagePk, pageUsageSk, PAGE_USAGE_HEAD_SK, usageGsiPk, usageCountGsiPk). See media § Usage list and delete safeguards for the port.
- Why a counter. A transaction cannot test that a partition is empty, so the cascade cannot check “no
PAGE#rows” directly. The counterncounts the asset’s rows and changes in the same transaction as every row it counts. Each such change also adds 1 to itsrev. The cascade’s transaction keeps the assetDeleteat index 0 and the name item at index 1, and adds at index 2 aConditionCheckon the counter:attribute_not_exists(pk) OR n = :zerofor a plain delete (deleteAssetanddeleteAssetCascadeboth refuse unless forced). A failed check reads the rows: any row isMediaInUseError; none is drift, retried within the delete’s 5 attempts and thenMediaConflictError('Usage count for asset <id> is above zero with no usage rows; run the usage rebuild'), never a silent delete. A forced cascade (usage: 'ignore') reads the counter, then the rows, and its index-2 check is “counter unchanged since read” (rev = :rev, or absent). The.usagerows it returns are therefore exactly the rows at the parent delete, even though the cascade is parent-first rather than one transaction. ATransactionConflict, or a forced cascade whose counter moved, is retried in the same loop with a jittered backoff; when the last failure was the counter moving, the error isMediaConflictError('Usage of asset <id> kept changing while it was deleted; try again'). - Writes (
replacePageUsage): read the page head and its mirror partition (consistent), diff, then send oneTransactWriteItemsper chunk of at mostUSAGE_ASSETS_PER_TRANSACTION= 20 assets. An added asset takes four items (aConditionCheckthatASSET#idexists in the page’s tenant; for aportfoliopage an asset item with notenantattribute, whichitemToAssetreads asportfolio, passes too, the counterADD n :1, the row and the mirror, bothattribute_not_exists), plus the head: 20 × 4 + 1 = 81, under DynamoDB’s 100-item limit. A removed asset is the counterADD n :-1(conditionedattribute_exists, so a missing counter is never created at -1) and the twoDeletes; a kept row that changed is the asset check and twoattribute_existsPuts; an unchanged kept row is only the asset check. Every chunk condition-checks and bumps the page headrev(ruling H2), so each chunk commits only against the state it planned from; writers of a multi-chunk page may alternate chunks, and the last to finish wins; a writer whose head is stale re-reads and re-diffs. A stale head or row, a counter condition or aTransactionConflictre-reads and re-diffs, up to 5 failed attempts with a jittered backoff, thenMediaConflictError. A page write is bounded: after its planned chunks, one asset-check retry per chunk, the 5 failed attempts and a small slack, it stops withMediaConflictErroreven if every commit succeeded (another writer kept undoing its work). - Reads.
listUsageForAsset:Query pk = USE#id AND begins_with(sk, 'PAGE#'), consistent.listUsageForPage: the mirror partition (begins_with(sk, 'USE#')), consistent. Both are sorted in code (byte order). removeAssetUsage: per chunk of 20 rows, delete each row (attribute_exists) and mirror and decrement the counter by the chunk size in the same transaction; delete the counter last, conditioned on no write since it was read (ruling M3). A counter that is missing is never re-created: the chunk carriesConditionCheck attribute_not_existsinstead. The counterDeleteretries aTransactionConflictException(3 jittered attempts, as in ruling L4).rebuildUsage: enumerate the tenant’s mirrors (gsi2onT#pf#USAGE, paged) and counters (gsi2onT#pf#USAGECOUNT). First repair the counter of every asset seen: read the counter (consistent), then count the rows (consistent), thenSET n = :countconditioned on the counter being unchanged since the read, retried on a failed condition (and aTransactionConflictException, 3 jittered attempts). Each counter that changes adds 1 tocountersRepaired. Then every listed page and every existing page of the listed types is written likereplacePageUsagewithliverows, except that a row younger thanPENDING_USAGE_GRACE_MSis neither removed nor rewritten (ruling H1). Counters are repaired before the pages, so a page write’s decrement finds its counter.- Conflicts with renames and meta edits. A usage write’s
ConditionCheckonASSET#idtakes part in DynamoDB’s conflict detection, so an overlapping rename (TransactWriteItemson the asset) orupdateAssetMeta(UpdateItem) can lose with aTransactionConflict. Both retry the same write up to 3 times with a jittered backoff (ruling L4;retryTransactionConflictsintransaction-errors.ts), then answerMediaConflictError(409). - IAM.
DynamoDBCrudPolicyonMediaTable(both the Nitro function and the media worker ininfra/sam/template.yaml) already grantsConditionCheckItem,UpdateItem,DeleteItem,PutItemandQueryon the table and its indexes; nothing changes. - Tests.
tests/dynamo-usage.spec.tsruns the usage contract on the in-memory DocumentClient plus the DynamoDB specifics (chunks, head, counter, repair).tests/dynamo-local.contract.spec.tsrunsdescribeUsageContract('dynamodb local', dynamoUsageHarness(endpoint))with a fresh table per case, and its own cases: the drift case, the 30-asset chunking case (each on its own table) and two cases that hold one side of the race at its transaction, so both branches run on the real engine (left alone, the cascade commits first). SetDYNAMODB_TEST_ENDPOINT(for example a throwawaydocker run -d -p 127.0.0.1:18010:8000 amazon/dynamodb-local:2.5.2).
Presign and S3 headers
createPresignedIngest({ path, assetId, mime, bucket, byteSize? }) signs a PUT for an already reserved key, at its private pending key pending/<assetId>/<path> (s3Key; media § Name reservation).
Clients. S3BlobStore, createPresignedIngest, invalidateMediaPaths and the SQS helpers share one client per configuration for the process (sharedS3Client, sharedCloudFrontClient, created on first use). Every media AWS client, the DynamoDB document client of createDynamoMediaStores included, starts from mediaClientConfig(region, endpoint) in src/client-config.ts (no SDK import, so a DynamoDB- or SQS-only path does not load the S3 and CloudFront clients), which applies MEDIA_AWS_CLIENT_DEFAULTS: maxAttempts: 3, a 3 s connection timeout and a 30 s socket idle timeout, so a stalled endpoint fails the request instead of hanging it. Pass client to use your own. With byteSize it also signs content-length (ContentLength, kept in the signed headers), so S3 refuses a body of any other length, and echoes byteSize in the result; the uploader must send exactly that many bytes. Without it no length is signed (the unsized expand phase, see media-nuxt § Upload size cap). Content-Length is not in PRESIGN_REQUIRED_HEADERS: browsers and fetch set it from the body. PRESIGN_REQUIRED_HEADERS (Content-Type, If-None-Match, Cache-Control, Content-Disposition) is the single list of headers the uploader must send, each with exactly the value returned in headers (they are signed into the URL); the bucket CORS AllowedHeaders must cover it, which scripts/sam-template.test.js checks by importing packages/media-aws/src/presign-headers.ts directly (Node type stripping, no build step; the file has no imports on purpose).
S3BlobStore.put sets ContentType, CacheControl and ContentDisposition when given (server-side writes: ingest and variants). Presigned uploads carry the same headers (Cache-Control: public, max-age=31536000, immutable, Content-Disposition: inline; filename="<name>.<ext>", the asset mime), so browser and admin remote-forward uploads match server-side writes.
put(key, data, { ifMatch }) sends IfMatch (instead of IfNoneMatch: '*' for createOnly): upload sanitizing rewrites a presigned object at its own key only if it is still the object it read; 412, 409 or 404 then throw MediaBlobChangedError, which upload complete reports as a 409 conflict. createAwsMediaLibrary({ sanitize }) takes the sanitizer; the web server passes the raster one (upload sanitizing) and the MediaWorker none, so svgo, pdf-lib and ffmpeg never ship in the worker (tests/worker-boundary.spec.ts walks the worker’s static and dynamic imports through the workspace packages and fails if any of them, or @tgmc/image/sanitize, is reachable).
copy(from, to, { createOnly, ...headers }) is a server-side CopyObject with the new object headers (MetadataDirective: REPLACE); createOnly sends If-None-Match: * on the CopyObject, so S3 refuses atomically to write over an existing target (a 412 / 409 becomes MediaBlobExistsError); a HeadObject first only skips the copy when the target is plainly there. Renames therefore never stream objects through the Lambda. getPrefix(key, length) is a ranged GetObject (an empty object answers 416, read as empty bytes), used by the upload markup check.
Variant replacement, rename and delete call invalidateMediaPaths (/m/… paths) when ASSETS_DISTRIBUTION_ID is set. IAM for the worker and Nitro grants m/* (and s3:ListBucket on prefix m/*, m/) in infra/sam/template.yaml.
Endpoint overrides (DynamoDB Local + MinIO)
S3BlobStoreOptions, DynamoMediaStoresOptions (endpoint), the presigner and createAwsMediaLibrary (dynamoEndpoint, s3Endpoint, s3ForcePathStyle) accept custom endpoints; forcePathStyle is needed for MinIO. createPresignedIngest also takes publicEndpoint: the URL is signed against the host the uploader can reach (http://localhost:4420 in docker:test), not the server’s own http://minio:9000. mediaAwsEndpointsFromEnv() maps AWS_ENDPOINT_URL_DYNAMODB, MEDIA_S3_ENDPOINT / AWS_ENDPOINT_URL_S3, MEDIA_S3_FORCE_PATH_STYLE=1 and MEDIA_S3_PUBLIC_ENDPOINT; the SQS worker uses it too. See media-nuxt and docker. scripts/media-smoke.js (npm run docker:test:smoke, CD with MEDIA_SMOKE_ENABLED=1) exercises presign → PUT → complete → on-ingest processing → materialize → delete end to end.
Upload triggers
On SAM stacks the on-ingest recipes (media § On-ingest recipes) run in the media worker, started by S3 rather than by complete (one owner; see media-nuxt § On-ingest processing).
- Template (
infra/sam/template.yaml):AssetsBucket.NotificationConfiguration.QueueConfigurationssendss3:ObjectCreated:*under prefixm/toMediaQueue.MediaQueuePolicyletss3.amazonaws.comsqs:SendMessage, conditioned onaws:SourceArnandaws:SourceAccount. The ARN is built from the same expression asBucketName(arn:${AWS::Partition}:s3:::${AWS::StackName}-assets-${AWS::AccountId}), never!GetAtt AssetsBucket.Arn: the bucketDependsOnthe policy (S3 sends ans3:TestEventwhen the notification is saved), so a reference back would be a cycle. Adding the notification is an in-place bucket update (no replacement;BucketNameandObjectLockEnabledare unchanged, Retain stays).MediaQueueandMediaDLQpin SSE-SQS (SqsManagedSseEnabled: true, an in-place update) instead of relying on the account default: S3 cannot publish to a queue encrypted with the AWS managedaws/sqsKMS key. The worker’s SQS event hasBatchSize: 1andScalingConfig.MaximumConcurrency: 5.scripts/sam-template.test.jspins all of this, including a dependency-cycle check overRef/GetAtt/Sub/DependsOn. - Messages (
parseMediaQueueMessage,s3-events.ts):{ jobId }(materialize),{ type: 'on-ingest', assetId }(re-run), and S3 event records.s3:TestEvent, non-ObjectCreatedrecords,@variant keys (the worker’s own writes land underm/too) and keys that are not originals are ignored. Keys are URL-decoded (+is a space); a record from another bucket is skipped. - Registered row. Presigned uploads land under the private
pending/prefix, outside the notification filter (an event for apending/key is ignored), so an upload’s event is theObjectCreated:Copyofcompletepublishing it tom/, just before it writes the row. That event can still arrive before the row exists. The worker looks the key up withgetPathReservation, waits about 15 s in-process while the reservation is still pending, then throws so SQS redelivers it (visibility 360 s). On the third receive — the one that would dead-letter it (maxReceiveCount: 3,S3_EVENT_MAX_RECEIVES) — it logs and gives up quietly; an object with no holder at all (a stray upload) is skipped at once. A row created more than 5 minutes before the event (IMPORTED_ROW_AGE_MS; amedia:transfercopy keeps itscreatedAt) is skipped, since the transfer brought its children. A deleted asset is acknowledged. - Retries. A retryable recipe report (an unreadable original, a failed variant) is retried by redelivery within the same bound; deterministic failures (undecodable image) are recorded in meta and acknowledged.
- Existing assets are not backfilled: S3 only notifies new objects. Open the asset in the admin and use Re-run processing.
- Concurrent runs. Two runs at the same moment (an S3 event and a Re-run click, or a duplicated S3 event) still end with one child per variant: catalog child ids are deterministic and the child
Putis create-only (attribute_not_exists(pk)), so the loser reports in progress (media § On-ingest recipes). - Known gap.
pdf-libis not bundled into the worker, sopageCountis skipped on Lambda (TODO).
Transfer (seed and backup)
transferMedia from @tgmc/media works on these adapters. createDynamoMediaStores plus S3BlobStore form one MediaEndpoint, and S3BlobStore.digestSha256 re-hashes each copied object. insertAsset writes the NAME# claim in the same transaction as the asset, and objects get the immutable cache and inline disposition headers. npm run media:transfer (scripts/media-transfer.js) wires aws:<table>@<bucket> endpoints with the same --dynamodb-endpoint / --s3-endpoint / --path-style overrides. See media § Transfer between backends and the production runbook.
Env (Lambda)
MEDIA_TABLE / NUXT_MEDIA_TABLE, ASSETS_BUCKET, MEDIA_QUEUE_URL, ASSETS_DISTRIBUTION_ID, ADMIN_TOKEN, NUXT_AWS_REGION.
When the data backend is not dynamodb (host npm run dev, or DATA_BACKEND=sqlite|postgres), media uses SQLite/FS under MEDIA_LOCAL_ROOT (default .data/media; sync materialize) even if the table and bucket are set, and presign / complete switch to local presign mode (a signed PUT to /api/media/_local-s3/<key> into FsBlobStore, same headers and checks; see media-nuxt).