diff --git a/docs/changelog/2026-10-02-c248.md b/docs/changelog/2026-10-02-c248.md new file mode 100644 index 00000000..3b8f2b73 --- /dev/null +++ b/docs/changelog/2026-10-02-c248.md @@ -0,0 +1 @@ +**C248 — Miniaturas AVIF pré-geradas do álbum público (previews rápidos nos cards) (2026-10-02):** a grade `/fotos` deixa de pagar resize por request: cada foto aprovada ganha uma derivada AVIF 720px q60 armazenada como **objeto-irmão determinístico** no mesmo store privado do original (`flickr-.` → `flickr--grade.avif`), servida direto pela rota `/fotos//midia?tamanho=grade` quando o cliente manda `Accept: image/avif` (token explícito; `*/*`/sem header segue no JPEG). A rota ganha a negociação pura (`acceptsAvif`, `Vary: Accept` mantendo `private, no-store` e `noindex`), um dono de I/O estendido em `privateMediaResponse` (`privateMediaObjectExists`/`writePrivateObject`/`encodePrivateImageFile`/`encodePrivateMediaImage`, este extraído do resize on-the-fly sem mudança de comportamento) e o heal preguiçoso: derivada ausente/ilegível cai no fallback JPEG de sempre e a geração é reagendada em `after()`; a aprovação aquece a derivada best-effort no `afterChange` (falha nunca bloqueia a publicação; catalogação/índice facial/C246 não disparam porque o status não muda). O backfill one-shot `pnpm archive:thumbnails` (plan read-only / `--apply` / `--verify`) é idempotente por HEAD, elegível = aprovadas com arquivo, classifica `generated | already-present | missing-origin | corrupt-origin | failed` com recibo em `data/archive/reports/`, guardas `ARCHIVE_THUMBNAILS_CONFIRM=1` + `TEQO_ENV` exato + S3 completo em alvo não-local, e reusa a MESMA função de geração do servidor; o débito do C246 foi quitado movendo `mapWithConcurrency` e o parser `--only` para `scripts/lib/cli.mjs` no segundo consumidor. Sem migration, sem UI, sem mudança de URL pública nem no pull-down; runbook §C248. Testes: unit do naming/negociação/plan-lib, int da geração/heal e da matriz da rota (AVIF presente → `image/avif`; ausente/sem `Accept` → JPEG; hit não agenda heal, miss agenda), e2e `frontendFotos` mantido no JPEG com `Vary` pinado. diff --git a/docs/ops/teqo-1313-deploy.md b/docs/ops/teqo-1313-deploy.md index 2b028e55..242abca1 100644 --- a/docs/ops/teqo-1313-deploy.md +++ b/docs/ops/teqo-1313-deploy.md @@ -1762,6 +1762,63 @@ uma foto rebaixada por engano volta a `approved` pela ficha no admin (com o canal de remoção configurado) ou pelo publish depois de o objeto inspecionar limpo. +## C248 — miniaturas AVIF do álbum público (backfill das derivadas) + +O comando `pnpm archive:thumbnails` gera a miniatura AVIF 720px q60 de cada foto +**aprovada** do acervo como um objeto-irmão determinístico no mesmo store +privado do original (`flickr-.` → `flickr--grade.avif`); nada de +banco, nada de original alterado. A rota `/fotos//midia?tamanho=grade` +serve a derivada direto quando o cliente manda `Accept: image/avif`; sem o +token, ou com a derivada ausente/ilegível, ela responde o JPEG on-the-fly de +sempre e regenera a derivada em background (`after()`). A aprovação de uma foto +no admin também aquece a derivada (best-effort; falha nunca bloqueia). O lote +cobre o que foi aprovado antes disso e o que o hook perder (CLI em lote, +restart). + +```bash +# no homeserver: +ssh homeserver +cd ~/stack +# 1) plano read-only: elegíveis do álbum aprovado e quantas derivadas faltam +docker compose --profile maintenance run --rm \ + -v /srv/archive-reports:/app/data/archive \ + teqo-1313-migrate pnpm archive:thumbnails +# 2) canário (exige TEQO_ENV + confirmação; as 4 S3_* no alvo): +docker compose --profile maintenance run --rm \ + -e TEQO_ENV=production -e ARCHIVE_THUMBNAILS_CONFIRM=1 \ + -v /srv/archive-reports:/app/data/archive \ + teqo-1313-migrate pnpm archive:thumbnails --apply --limit 20 +# 3) lote completo (encode AVIF local; ~6,5k fotos): +docker compose --profile maintenance run --rm \ + -e TEQO_ENV=production -e ARCHIVE_THUMBNAILS_CONFIRM=1 \ + -v /srv/archive-reports:/app/data/archive \ + teqo-1313-migrate pnpm archive:thumbnails --apply +# 4) aceite: cobertura 100% (read-only; sai 1 se faltar alguma): +docker compose --profile maintenance run --rm \ + -v /srv/archive-reports:/app/data/archive \ + teqo-1313-migrate pnpm archive:thumbnails --verify +# 5) conferir uma resposta real negociada (espera content-type image/avif): +curl -sD - -o /dev/null -H 'Accept: image/avif' \ + "https://jorgesolla1313.com.br/fotos//midia?tamanho=grade" +``` + +Guardas: fora de teste o `--apply` exige `TEQO_ENV` casando o **nome exato** do +banco (`teqo_staging`/`teqo_1313`), `ARCHIVE_THUMBNAILS_CONFIRM=1` e — em alvo +não-local — as 4 `S3_*` (o original vive no bucket); `ALLOW_REMOTE_DB` é +recusado. O plano e o `--verify` são read-only. Reexecutar converge: derivada +existente é pulada por HEAD; original ausente/ilegível vira "pulada" com o +motivo no recibo (a C246 recupera o original e um novo `--apply` regenera); +falha de rede/upload fica registrada como falha e retenta na próxima execução. +**Serialize com `archive:catalog` e `archive:integrity`** (C245/C246 tocam os +mesmos originais) e rode em janela própria. `--concurrency` default 2 (o mesmo +teto do processo web), `--only`/`--limit` são os canários; recibos em +`data/archive/reports/`. + +Nada a revalidar depois do lote: a rota descobre a derivada por chave +determinística a cada request (`force-dynamic`, `no-store`). Rollback: sem +migration nesta entrega; basta remover os objetos `flickr--grade.avif` do +bucket (a rota volta ao resize on-the-fly sozinha) — o original nunca é tocado. + ## C230-followup — importação do Instagram para a Central de Conteúdos O comando `pnpm content:instagram:import` traz as mídias recentes do perfil diff --git a/docs/plans/album-miniaturas-avif-impl.md b/docs/plans/album-miniaturas-avif-impl.md new file mode 100644 index 00000000..3d4f562d --- /dev/null +++ b/docs/plans/album-miniaturas-avif-impl.md @@ -0,0 +1,145 @@ +# Impl: C248 — Miniaturas AVIF pré-geradas do álbum público (previews rápidos nos cards) + +Status: aprovado +Atualizado em: 2026-10-02 +Issue: #1424 +Intenção: docs/plans/album-miniaturas-avif.md +Appetite restante: herdado (~1–2 dias eng + máquina do backfill; cortes: sem UI, sem migration, sem variante do lightbox, sem CDN/cache novo, sem tocar no original) + +## Leitura da intenção + +- **Outcome:** a grade pública (`?tamanho=grade`) deixa de pagar sharp por request quando existe a derivada AVIF 720px q60 armazenada; qualquer ausência/falha cai no fallback on-the-fly atual, nunca em imagem quebrada; navegador sem AVIF recebe JPEG; pull-down imediato preservado; fotos novas cobertas pelo fluxo de aprovação + heal; backfill one-shot idempotente com recibo e guarda `*_CONFIRM`. +- **O que NÃO negociar:** URL congelada `/fotos//midia?tamanho=grade`; `private, no-store`; bucket privado Garage sem CDN; pull-down imediato (o gate continua sendo o read cacheado `getApprovedArchivePhotoById`); fallback sempre que a derivada faltar; AVIF q60 na régua do repo; filename determinístico por foto; nenhum original alterado; nenhuma URL/UI nova. +- **O que reavaliar (achados confirmados no código):** + - (a) A rota já lê a linha fresca (`findByID`, `route.ts:46-54`) para filename/mime — é esse filename que deriva a chave da derivada; nenhuma query nova. + - (b) O resize on-the-fly vive no dono `buildPrivateMediaImageResponse` (`privateMediaResponse.ts:230-296`), que já abre o objeto pelo caminho S3/disco (`openPrivateObject`) — a derivação reusa esse dono, não cria segundo pipeline de I/O. + - (c) Não existe negociação de imagem (`Accept`/`Vary`) em lugar nenhum do repo; o default seguro é JPEG e o Playwright (`Accept: */*`) mantém o e2e atual verde. + - (d) A régua AVIF q60 mora em `scripts/lib/imageResize.mjs:39` (`.mjs`, não importável de `src/`): as constantes vão para `lib/` com unit pin e comentário do dono. + - (e) `afterChange` fire-and-forget pode morrer no `process.exit` das CLIs (ex.: `archive:publish` aprova em lote num processo efêmero) — o heal preguiçoso na rota é a garantia; o hook é warm-up. + - (f) C246 re-ingere o original na MESMA chave (mesma foto, `repairArchivePhotoObject`) e a C245 só escreve metadados — a chave determinística da derivada sobrevive; objeto ausente/corrupto é skip honesto e o heal regenera quando recuperado. + +## Abordagem recomendada + +```mermaid +flowchart TD + A["GET /fotos//midia?tamanho=grade"] --> B{"gate: getApprovedArchivePhotoById (read cacheado)"} + B -- "não aprovada" --> Z["404 silencioso (no-store, noindex)"] + B -- aprovada --> C{"download=1?"} + C -- sim --> D["original / download (como hoje)"] + C -- não --> E{"acceptsAvif(Accept)?"} + E -- "não (default, */*)" --> F["fallback: sharp → JPEG q80 on-the-fly (dono privateMediaResponse)"] + E -- sim --> G["tenta flickr--grade.avif (buildPrivateMediaResponse, image/avif)"] + G -- "404/erro/ilegível" --> H["after(): ensureArchivePhotoGrade — heal preguiçoso"] --> F + G -- "200" --> I["serve a derivada armazenada"] + F --> J["Vary: Accept + private, no-store + X-Robots-Tag: noindex"] + I --> J + K["afterChange: transição PARA approved"] --> L["ensureArchivePhotoGrade (best-effort, não bloqueia)"] + M["pnpm archive:thumbnails --apply"] --> N["generateArchivePhotoGrade (mapWithConcurrency)"] + L --> O[("bucket privado / staticDir: flickr--grade.avif")] + N --> O + H --> O +``` + +**Opções consideradas:** A) derivada como objeto-irmão determinístico no mesmo store privado, descoberta no caminho da rota, com extensão do dono `privateMediaResponse` | B) `imageSizes`/variante do Payload na collection `archivePhoto` (com backfill de regeneração) | C) segundo pipeline de mídia (fila/CDN/collection própria). +**Recomendação:** A — o bucket/disco privado já é o único store do original, o filename do original já é determinístico e imutável (C246 sobrescreve a mesma key), e o dono do I/O privado (`privateMediaResponse`) já concentra abrir/escrever/encodar; a derivada vira um irmão de chave (`flickr-.jpg` → `flickr--grade.avif`), sem schema nem segunda fonte de verdade. Um HEAD a mais no caminho da derivada substitui o decode do original no caminho atual (hoje: HEAD+GET+sharp; com derivada: HEAD+GET). +**Rejeitadas:** B porque `imageSizes` só gera no upload, exigiria regeneração/backfill acoplado ao Payload e criaria estado de linha para um objeto derivado (migration desnecessária); C porque não existe fila/worker no stack self-hosted e CDN/cache novo é anti-goal explícito. + +### Componentes / mudanças + +- **`archivePhotoThumbnail`** (`src/lib/archivePhotoThumbnail.ts`, novo — puro, client-safe): `ARCHIVE_PHOTO_GRADE_EXTENSION`/`ARCHIVE_PHOTO_GRADE_MIME_TYPE` (`image/avif`)/`ARCHIVE_PHOTO_GRADE_QUALITY` (60, régua `scripts/lib/imageResize.mjs:39`) e `archivePhotoGradeFilename(originalFilename)` — irmão determinístico (`flickr-.jpg` → `flickr--grade.avif`); largura reusa `ARCHIVE_PHOTO_THUMB_WIDTH` (720, `archivePhotoPublicCatalog.ts:43`). Compartilhado por rota, utility de geração, hook e script. +- **`acceptsAvif`** (`src/lib/privateMedia.ts`, edição): negociação pura do header — token exato `image/avif` (case-insensitive, `q` ausente ou > 0); `*/*`, `image/*` e header ausente ⇒ falso (default JPEG). Unit-pinada no spec existente. +- **`privateMediaObjectExists` / `writePrivateObject` / `encodePrivateMediaImage`** (`src/utilities/privateMedia/privateMediaResponse.ts`, edição): extensões do dono do I/O privado. `privateMediaObjectExists` = HEAD S3 / `stat` local, nunca lança; `writePrivateObject` = `PutObjectCommand` (ContentType) no S3 ou temp+rename no `staticDir` local (mesmo guard de path do `resolveLocalPath`); `encodePrivateMediaImage({ media, staticDir, width, format, quality })` extrai o stream→sharp (`failOn:'error'`, `rotate`, resize `withoutEnlargement`) que hoje é inline no fallback, e `buildPrivateMediaImageResponse` passa a chamá-lo com `jpeg q80` (comportamento idêntico, unit/e2e pinam). +- **`archivePhotoThumbnails`** (`src/utilities/archivePhotos/archivePhotoThumbnails.ts`, novo — `server-only`): `generateArchivePhotoGrade(payload, { filename })` baixa o original com `downloadPrivateMediaToFile`, encoda AVIF 720 q60 (`encodePrivateImageFile`) e grava com `writePrivateObject`; classifica `generated | skipped (missing-origin | corrupt-origin | no-filename) | failed`; nunca lança. `ensureArchivePhotoGrade(payload, { filename })` adiciona probe (`privateMediaObjectExists`), in-flight Map, limite de concorrência 2 e memo de falha 5 min — molde `contentPieceFrameJob` (`contentPieceFrameJob.ts:74-105,249-283`). Sem advisory lock/transação: não escreve DB e a chave é idempotente (last-write-wins do mesmo conteúdo). +- **Rota** (`src/app/(frontend)/fotos/[id]/midia/route.ts`, edição): mantém id/gate/`findByID`/download; em `tamanho=grade` sem `download` e `acceptsAvif(request.headers.get('accept'))`, tenta a derivada (`buildPrivateMediaResponse` com `{ filename: gradeName, mimeType: image/avif }`, `rangeHeader: null`) e, em 404/erro, cai no fallback atual e agenda `after(() => ensureArchivePhotoGrade(...))`; `Vary: Accept` na resposta da grade (derivada e fallback). +- **Hook** (`src/collections/ArchivePhoto.ts`, edição): `warmArchivePhotoGradeAfterChange` no `afterChange` — só na transição PARA `approved` (`previousDoc?.publicationStatus !== 'approved'`) com `filename` → `void ensureArchivePhotoGrade(...).catch(() => undefined)`; cataloguing/face-index/C246-repair não disparam (status não muda); falha jamais impede a aprovação. +- **Backfill** (`scripts/backfill-archive-thumbnails.mjs` + `scripts/lib/archiveThumbnailPlan.mjs`, novos; script `"archive:thumbnails"` no `package.json`): plan default read-only, `--apply` (gera faltantes), `--verify` (cobertura); `--only/--limit/--concurrency/--out/--help`; reusa `generateArchivePhotoGrade`, `privateMediaObjectExists`, `mapWithConcurrency` e `writeRepoFile`; recibo JSON em `data/archive/reports/` (gitignored) com processadas/puladas/falhas. +- **Pins** (`scripts/lib/test-affected-core.mjs`, edição): `scripts/lib/archiveThumbnailPlan.mjs` em `SCRIPTS_SPEC_PINNED` (o unit spec importa; `ciSkipInvariants.unit.spec.ts` exige a igualdade exata). O manifest e2e já cobre os prefixos tocados (`src/lib/archivePhoto`, `src/utilities/archivePhotos`, `src/utilities/privateMedia`, `src/collections/ArchivePhoto.ts`, `src/app/(frontend)/fotos`) → `frontendFotos`/`frontendFotosSelfie`; sem entrada nova. +- **Testes** (edição/novos): unit `tests/unit/archivePhotoThumbnail.unit.spec.ts` (naming/constantes) + extensão de `tests/unit/privateMedia.unit.spec.ts` (`acceptsAvif`) + `tests/unit/archiveThumbnailPlan.unit.spec.ts`; int extensão de `tests/int/archivePhotoPublicRead.int.spec.ts` (matriz da rota) + `tests/int/archivePhotoThumbnails.int.spec.ts` (geração/ensure/heal); e2e `tests/e2e/frontendFotos.e2e.spec.ts` (só `Vary`; JPEG mantido). +- **Runbook** (`docs/ops/teqo-1313-deploy.md`, edição): seção C248 (canário, staging→produção, `--verify`); changelog `docs/changelog/2026-10-02-c248.md` no fechamento. +- **Migration:** nenhuma — a derivada é objeto no store privado, sem campo/índice/collection; a chave deriva do `filename` que a rota já lê (stop-condition: surgir necessidade de linha/marcador ⇒ parar e sinalizar). +- **Access / Consent:** nenhuma mudança — a rota segue com o gate público cacheado e o `overrideAccess` documentado (releitura de filename); a geração é escrita de objeto por ator confiável já autorizado (mesmas credenciais S3 do ingest, sem permissão nova); sem `Consent`/`Contact`. +- **UI:** Impeccable A — nenhuma tela/fluxo; zero dispatch de designer; o card segue com o mesmo `` (`ArchivePhotoCard.tsx:29`), que negocia `Accept` sozinho. + +### Dados → forma + +- Sem dados de produto novos (a grade e o overlay não mudam). Único artefato: o recibo operacional do lote em `data/archive/reports/archive-thumbnails-.json` — `{ runAt, mode: plan|apply|verify, target, eligible, generated, skipped[], failed[], results[], durationMs }` consumido por operador. +- Rejeitadas: painel/admin (não pedido), CSV (ninguém consome), segunda fonte de contagem (o `--verify` recomputa do HEAD, que é a verdade). + +## Decisões de engenharia + +**D1 — Chave/nome da derivada e descoberta (caro)** + +- Opções: A) objeto-irmão no mesmo store privado, chave derivada do `media.filename` (`flickr--grade.avif`), descoberta por tentativa de GET/HEAD no caminho da rota, miss ⇒ fallback; B) linha no DB (segunda linha em `archivePhoto` ou collection nova) com filename determinístico, descoberta por query; C) prefixo/pasta separada (`grade/...`). +- Recomendação: A — o original já tem nome determinístico e imutável (`archivePhotoStorageFilename`, `archivePhoto.ts:91-104`), o C246 sobrescreve a MESMA key, e a rota já tem o filename fresco em mãos; não há migration, não há estado a dessincronizar e o HEAD extra substitui (não soma ao) o HEAD que o caminho atual já faz no original. Miss = 404 do objeto → fallback on-the-fly + heal (D3); derivada ilegível (erro de leitura/checksum) também cai no fallback. +- Rejeitadas: B — migration/linha órfã, derivadas apareceriam nas listas/select da collection e o probe viraria query por um objeto que o bucket já sabe responder; C — hierarquia nova sem ganho (o espaço de chaves já é flat e o `resolveLocalPath` local teria que criar subdir), com risco de colisão de prefixo em varreduras futuras. + +**D2 — Contrato de negociação de formato (caro)** + +- Opções: A) negociar por `Accept: image/avif` explícito (token exato, `q > 0`), `Vary: Accept`, default JPEG quando o header falta ou não pede AVIF; B) servir AVIF sempre que existir (exige `` na UI para o fallback); C) parâmetro de URL (`?formato=avif`). +- Recomendação: A — cumpre "nenhum preview quebrado" sem UI, e `private, no-store` torna o `Vary` cinto de segurança (nenhum cache compartilhado guarda a resposta, mas intermediários heurísticos não podem misturar). Requerer o token explícito é o default fail-safe: Playwright/API clients (`*/*`) continuam no JPEG e o e2e existente não muda; browsers modernos mandam `image/avif` e ganham a derivada. +- Rejeitadas: B — browser sem decodificador AVIF mostraria imagem quebrada e a correção seria ``/UI, fora do escopo; C — cria contrato/param novo em URL congelada. + +**D3 — Seam da geração automática (caro)** + +- Opções: A) `afterChange` aguardando a geração (bloqueia a aprovação); B) híbrido: `afterChange` fire-and-forget na transição PARA `approved` + heal preguiçoso na rota no miss (`after()`) + backfill one-shot; C) só heal preguiçoso na rota (sem hook e sem backfill); D) fila/job novo. +- Recomendação: B — o hook cobre a aprovação interativa (foto nova já nasce com derivada, sem custo no request de preview); o heal preguiçoso é a garantia para os casos em que o fire-and-forget morre (CLI de publicação em lote faz `process.exit`, restart de deploy) e para fotos recuperadas pela C246; o backfill aquece as ~6.492 aprovadas existentes. Falha de geração nunca lança e nunca impede a aprovação (o fallback atende). +- Rejeitadas: A — acopla a latência da aprovação ao encode AVIF (1–3s/foto; lote inviável) e transformaria bug de encode em erro de publicação; C — não cumpre o aceite "cobertas automaticamente pelo fluxo de entrada/aprovação" e deixa as 6.492 reféns do primeiro visitante, sem recibo de backfill; D — não existe worker/queue no stack e criar um seria pipeline genérico (anti-goal). + +**D4 — Lote/backfill, guardas e recibo (caro)** + +- Opções: A) CLI `pnpm archive:thumbnails` com plan-lib puro, modos plan/`--apply`/`--verify`, `ARCHIVE_THUMBNAILS_CONFIRM`, recibo JSON e a MESMA função de geração do servidor; B) endpoint/admin action para disparar o lote; C) script inline sem lib pura/testes. +- Recomendação: A — segue o padrão C245/C246 (`assertWriteConfirm` + `assertEnvironmentDatabaseTarget` + `mirroredMediaRequired`, recibo em `data/archive/reports/`, skip honesto); elegível = `publicationStatus === 'approved'` com `filename` (produto Q3); idempotente por HEAD (já existe = "pulada"; ausente/corrupto na origem = "pulada" com motivo; falha de rede/upload = "falha" e a próxima execução retenta); concorrência default 2 (o teto real do encode por processo, o mesmo slot do web) com `--only/--limit` de canário; `mapWithConcurrency` e o parser `--only` compartilhados movidos para `scripts/lib/cli.mjs` no 2º consumidor (débito S3 do C246 quitado neste PR). +- Rejeitadas: B — superfície de operação de máquina sem recibo/guarda, e a rota pública não deve disparar lote; C — quebra o padrão de plan-lib testável e o pin do CI. +- **Serialização C245/C246:** o backfill roda serializado com `archive:catalog`/`archive:integrity` (nota no runbook); original ausente/corrupto = skip com motivo, regenerado quando o C246 recuperar (heal ou novo `--apply`); nenhum lock novo — a chave é idempotente e o pior caso é trabalho redundante. + +**D5 — Testes e o que muda no e2e existente (barato, mas explícito)** + +- Opções: A) unit do contrato puro + int da rota/geração (storage local) + e2e existente mantendo JPEG e ganhando só o `Vary`; B) só int; C) e2e novo do AVIF gerado ponta-a-ponta. +- Recomendação: A — o naming e a negociação são contrato puro (unit); a matriz da rota (derivada presente + `Accept` avif → 200 `image/avif`/`Vary`/`no-store`; derivada ausente → 200 `image/jpeg`; sem `Accept` → JPEG; draft/removida → 404) e a geração no `staticDir` local (gera, segunda chamada pula, origem corrompida = skip sem derivada) são determinísticas em int; o e2e `frontendFotos` já pinava `content-type: image/jpeg` (`frontendFotos.e2e.spec.ts:359`) porque o APIRequestContext manda `*/*` — a expectativa fica e só se acrescenta `vary: Accept` (determinístico). +- Rejeitadas: B — deixa chave/negociação sem pin de contrato; C — o hook é fire-and-forget e o heal é assíncrono: e2e do AVIF gerado viraria poll/flake; a cobertura fiel é int (função direta) + e2e do fallback. + +## Fases verificáveis + +1. **Tracer/servidor — quota ~45%:** `lib/archivePhotoThumbnail` + `acceptsAvif` + extensões do dono `privateMediaResponse` + `archivePhotoThumbnails` + rota + hook. Testes: unit do naming/negociação; int da rota (matriz acima, mockando `next/server` `after` como `contentPiece.int.spec.ts:21-26`) e da geração local. Prova local (tracer bullet): aprovar UMA foto seedada → derivada nasce no `staticDir`; `Accept: image/avif` → 200 avif; sem header → 200 jpeg; remover a derivada → fallback e regeneração agendada. +2. **UI (n/a) — quota 0%:** sem tela/fluxo novo, sem dispatch de designer (Impeccable A); o card não muda. +3. **Lote + runbook — quota ~30%:** `scripts/lib/archiveThumbnailPlan.mjs` + `scripts/backfill-archive-thumbnails.mjs` + `"archive:thumbnails"` + pin `SCRIPTS_SPEC_PINNED` + unit do plan-lib + seção no `teqo-1313-deploy.md`. Prova local: `pnpm archive:thumbnails` (plan, read-only) lista elegíveis; `--apply --limit 5` gera e o recibo conta; reexecutar converge ("puladas"); `--verify` fecha. +4. **Gates — quota ~25%:** `pnpm gate:fast`; `pnpm test:int`; e2e local `pnpm test:e2e --no-deps --project=frontendFotos`; `pnpm push`; changelog C248 no fechamento. **Ops (fora do merge):** no homeserver, canário `--limit` em staging → lote em produção pelo serviço maintenance (env do stack) → `--verify`; anexar recibo à Issue #1424. + +## Rabbit holes / Não escopo (engenharia) + +- Variante média do lightbox, matriz de tamanhos/formatos (WebP/JPEG armazenados), tuning de qualidade/effort — só 720px AVIF q60 da grade. +- `imageSizes` do Payload, regeneração de variantes, collection/tabela de derivadas, migration, marcador no DB ou no admin. +- CDN/cache público, `s-maxage`/`immutable`, service worker, `next/image`, `` ou qualquer mudança de UI. +- Tocar no original, no `download=1` (segue original/nome legível), no range do original ou no gate/pull-down. +- Reaper/GC de objetos órfãos, invalidação proativa pós-C246, backfill concorrente ilimitado, fila/job genérico de mídia. +- Nova env de runtime (a chave da derivada é derivada; nada novo para o build) e importmap (sem componente admin). + +## Riscos e mitigação + +- **CPU de AVIF no processo web (heal por visita):** limite de concorrência 2 + memo de falha 5 min + `after()` (nunca na latência da resposta) + backfill como aquecimento antes de anunciar; medir p95 da grade localmente. +- **Fire-and-forget perdido (CLI/restart):** heal preguiçoso na rota cobre na primeira visualização; backfill cobre em lote; recibo mostra o que ficou pendente. +- **Derivada parcial no disco local (dev/test):** `writePrivateObject` grava temp + rename atômico no mesmo diretório. +- **Derivada corrompida/desatualizada:** qualquer erro de leitura cai no fallback; C246 sobrescreve o original da MESMA foto (conteúdo visual equivalente a 720px) e a regeneração só acontece se a chave sumir — o pior caso é JPEG on-the-fly, nunca imagem quebrada. +- **Browser sem AVIF / cache misto:** `acceptsAvif` exige token explícito (`*/*` ⇒ JPEG), `Vary: Accept` + `private, no-store` mantidos; e2e pin. +- **Backfill vs C245/C246:** rodar serializado (runbook); ausente/corrupto = skip com motivo; reexecução converge; `removed`/`draft` ficam fora da fila. +- **Credenciais S3 sem PutObject:** a geração usa o mesmo cliente/bucket que o ingest já escreve (Payload upload) — sem permissão nova; falha de permissão vira `failed` no recibo, sem afetar a rota (fallback). +- **Lote de ~6.5k lento:** concorrência 2, canário `--only/--limit`, recibo com duração/bytes; execução noturna no homeserver (sem egress externo). +- **CI:** a edição em `src/collections/ArchivePhoto.ts` e `package.json` é high-risk ⇒ e2e curado (`frontendFotos` já mapeado); `scripts/lib/archiveThumbnailPlan.mjs` entra no pin exato exigido por `ciSkipInvariants`. + +## Débitos diferidos (gatilhos) + +- **Corrida probe→in-flight no `ensureArchivePhotoGrade`** (débito do simplify): o probe HEAD precede o check de in-flight (molde C226), então um joiner cujo HEAD falhou pode re-encodar uma derivada recém-gerada. **Gatilho:** geração duplicada observada (log/contador) ou no lote de 6.5k. +- **Sem e2e do AVIF gerado ponta-a-ponta** (D5): cobertura int direto + e2e do fallback. **Gatilho:** regressão AVIF em produção ou runner estável para poll sem flake. +- **`privateMediaObjectExists` engole erro como "false"**: no `--verify`, bucket inacessível lê como "faltando" (falha fechada, sem verde falso); distinguir `unreachable` no recibo é polish. **Gatilho:** diagnóstico confuso em run real do `--verify`. +- **`--concurrency` do CLI capado em 2** (slot global do processo web): limitante documentado do throughput do lote. **Gatilho:** primeiro run real do lote lento demais → rever o slot por contexto (web vs CLI). + +## Aceite de engenharia + +- [ ] Aceite de produto coberto: derivada servida direto quando existe e o cliente aceita AVIF; fallback JPEG em ausência/erro/header sem AVIF; pull-down imediato pelo mesmo gate cacheado; fotos novas cobertas (hook + heal) e acervo existente pelo backfill; idempotência com recibo; nenhum original/URL/UI alterados. +- [ ] Invariantes: `lib` puro → `utilities` server-only → `app`; sem migration; `overrideAccess` só nos precedentes documentados (releitura de filename; script ator confiável); guardas `*_CONFIRM` + `TEQO_ENV` exato + S3 completo fail-closed; identifiers em inglês / copy pt-BR; nenhuma escrita no DB de produção pelo agente. +- [ ] Testes de domínio previstos (unit/int) onde o read/write path muda: naming/negociação puros; rota (avif presente/ausente/sem accept/404) e geração/heal em int com storage local; e2e `frontendFotos` mantém JPEG e ganha `Vary`. +- [ ] `pnpm gate:fast` verde; int verde; e2e `frontendFotos` local verde; pin/manifest conferidos; `pnpm push`; runbook + changelog no PR. + +## Self-score decision-quality + +**5/5** — (1) as decisões caras foram fechadas com opções/recomendação/rejeitadas: D1 chave/descoberta, D2 contrato Accept/Vary, D3 seam da geração, D4 shape/guardas do lote, D5 testes/e2e; (2) depth check — reusa `privateMediaResponse` (I/O + resize), `imageResize` (régua q60 via pin), guardas de `cli.mjs` e o molde `contentPieceFrameJob`/`recover-media`/`archiveIntegrityPlan`, sem abstraction nova com 1 call site e sem twin de pipeline; (3) a hipótese da intenção foi corrigida contra o código (filename fresco já disponível, `after()` indisponível no CLI ⇒ heal é a garantia, régua q60 mora em `.mjs`); (4) rabbit holes cortados (matriz de formatos, CDN, `imageSizes`, UI, GC de órfãos); (5) riscos com mitigação verificável (concorrência/memo, temp+rename, skip honesto, e2e mantido) e aceite com checkboxes. diff --git a/package.json b/package.json index 0672c46f..a1005039 100644 --- a/package.json +++ b/package.json @@ -44,6 +44,7 @@ "archive:publish": "cross-env NODE_OPTIONS=\"--no-deprecation --import=tsx/esm --import=./scripts/seed-loader.mjs\" node scripts/publish-archive-photos.mjs", "ops:global": "cross-env NODE_OPTIONS=\"--no-deprecation --import=tsx/esm --import=./scripts/seed-loader.mjs\" node scripts/ops-global.mjs", "archive:integrity": "cross-env NODE_OPTIONS=\"--no-deprecation --import=tsx/esm --import=./scripts/seed-loader.mjs\" node scripts/check-archive-integrity.mjs", + "archive:thumbnails": "cross-env NODE_OPTIONS=\"--no-deprecation --import=tsx/esm --import=./scripts/seed-loader.mjs\" node scripts/backfill-archive-thumbnails.mjs", "seed:face-consents": "cross-env NODE_OPTIONS=\"--no-deprecation --import=tsx/esm --import=./scripts/seed-loader.mjs\" node scripts/seed-face-consents.mjs", "seed:face-figures": "cross-env NODE_OPTIONS=\"--no-deprecation --import=tsx/esm --import=./scripts/seed-loader.mjs\" node scripts/seed-face-figures.mjs", "db:seed:minimal": "cross-env NODE_OPTIONS=\"--no-deprecation --import=tsx/esm --import=./scripts/seed-loader.mjs\" node scripts/seed-minimal.mjs", diff --git a/scripts/backfill-archive-thumbnails.mjs b/scripts/backfill-archive-thumbnails.mjs new file mode 100644 index 00000000..e9b3a0a6 --- /dev/null +++ b/scripts/backfill-archive-thumbnails.mjs @@ -0,0 +1,264 @@ +/** + * C248 — backfill of the stored grade thumbnails of the public album: one + * 720px AVIF q60 per APPROVED archive photo, written as a sibling object of + * the stored original (same key space, derived name), so the public grid + * serves it directly instead of resizing on every request. + * + * Read-only by default; `--apply` generates the missing grades and `--verify` + * checks the coverage of the approved album. The generation is the SAME + * function the route's lazy heal and the approval hook call, so the lote and + * the serving path can never drift. Re-running converges: an existing grade + * is skipped by HEAD, a missing/corrupt original is an honest skip with its + * reason and a network/upload failure is a named failure for the next run. + * + * Safety model (fail-closed, mirrors `archive:integrity`): + * - --apply requires the explicit intent flag ARCHIVE_THUMBNAILS_CONFIRM=1 + * on a non-local/production target and the declared TEQO_ENV matching the + * exact database name; a remote target also requires the complete S3_* set + * (the original lives in the private bucket there). + * - Only APPROVED photos with a stored file are eligible: draft/removed rows + * are out (generating for them would be discarded work) and neither the + * original nor any row is ever written. + * - Serialize with `archive:catalog` / `archive:integrity` (the C245/C246 + * ops touch the same originals): a missing original is regenerated after + * the C246 recovery with a new `--apply`. + * + * Modes: + * pnpm archive:thumbnails plan — lists the eligible album and + * how many grades are missing (HEAD) + * pnpm archive:thumbnails --apply generates the missing grades + * pnpm archive:thumbnails --verify coverage acceptance check + * + * Options: --only , --limit (canary), --concurrency (default 3), + * --out (receipts, default `data/archive`), --help. + * + * Runbook (production): on the homeserver, inside the compose maintenance + * service with the stack env — see docs/ops/teqo-1313-deploy.md §C248. + */ +import { join } from 'node:path' + +import { getPayload } from 'payload' + +import { + ARCHIVE_THUMBNAIL_DEFAULT_CONCURRENCY, + ARCHIVE_THUMBNAIL_DEFAULT_OUT_DIR, + archiveThumbnailEligibleRows, + archiveThumbnailReportStamp, + formatArchiveThumbnailProbeReport, + formatArchiveThumbnailRunReport, + parseArchiveThumbnailCliArgs, + summarizeArchiveThumbnailProbe, + summarizeArchiveThumbnailRun, +} from './lib/archiveThumbnailPlan.mjs' +import { + TEQO_ENV_DATABASE_BY_ENV, + assertEnvironmentDatabaseTarget, + assertWriteConfirm, + databaseTarget, + dieWithLabel, + loadCliEnv, + mapWithConcurrency, + mirroredMediaRequired, + writeRepoFile, +} from './lib/cli.mjs' + +loadCliEnv() + +const die = dieWithLabel('archive:thumbnails') + +const WRITE_CONFIRM_FLAG = 'ARCHIVE_THUMBNAILS_CONFIRM' + +const envLines = Object.entries(TEQO_ENV_DATABASE_BY_ENV) + .map(([environment, database]) => `${environment} (${database})`) + .join(' | ') + +const HELP = ` +Uso: pnpm archive:thumbnails [opções] + +Gera as miniaturas AVIF 720px q60 do álbum público somente para fotos +APROVADAS com arquivo armazenado — o irmão determinístico +flickr--grade.avif no mesmo store privado do original (nada de banco, +nada de original alterado). Reexecutar converge: miniatura existente é +pulada por HEAD; original ausente/ilegível é pulada com motivo; falha de +rede/upload fica registrada para a próxima execução. + +Modos (mutuamente exclusivos; o default é o plano): + (sem flag) plano read-only — lista elegíveis e quantas faltam + --apply gera as miniaturas faltantes (escrita só de objetos) + --verify checagem de cobertura (exit 1 se faltar alguma) + +Opções: + --only restringe a ids de foto separados por vírgula (ex.: --only 80,83) + --limit --apply: processa no máximo n fotos (canário); no plano/verify, limita a checagem + --concurrency workers simultâneos (default ${ARCHIVE_THUMBNAIL_DEFAULT_CONCURRENCY}) + --out diretório dos recibos (default ${ARCHIVE_THUMBNAIL_DEFAULT_OUT_DIR}) + --help esta ajuda + +Ambiente: + DATABASE_URL alvo (obrigatório em todos os modos) + S3_* obrigatório no --apply de alvo não-local (o original vive no bucket) + ARCHIVE_THUMBNAILS_CONFIRM=1 intenção explícita de escrita em alvo não-local/produção + TEQO_ENV=${envLines} casa o banco exato no --apply + +Serialização: rode em janela própria, sem archive:catalog / archive:integrity +concorrentes (C245/C246 tocam os mesmos originais). Após um --apply em produção, +nada a revalidar: a rota descobre a miniatura por chave determinística. +` + +const writeReport = async (options, report, suffix = '') => { + const stamp = archiveThumbnailReportStamp(report.runAt) + const relativePath = join(options.out, 'reports', `archive-thumbnails-${stamp}${suffix}.json`) + await writeRepoFile({ + label: 'archive:thumbnails', + root: process.cwd(), + relativePath, + body: JSON.stringify(report, null, 2), + }) + return relativePath +} + +async function main() { + const options = parseArchiveThumbnailCliArgs(process.argv.slice(2)) + if (options.help) { + console.log(HELP) + process.exit(0) + } + + if (!process.env.DATABASE_URL) die('DATABASE_URL não definida; recusando continuar.') + const mode = options.apply ? 'apply' : options.verify ? 'verify' : 'plan' + const runAt = new Date().toISOString() + console.log(`[archive:thumbnails] alvo: ${databaseTarget()} | modo: ${mode}`) + + if (mode === 'apply') { + assertWriteConfirm({ + label: 'archive:thumbnails', + flag: WRITE_CONFIRM_FLAG, + command: 'pnpm archive:thumbnails --apply', + }) + assertEnvironmentDatabaseTarget() + + const { resolveS3StorageEnv } = await import('../src/utilities/mediaStorage.ts') + const storage = resolveS3StorageEnv(process.env) + if (mirroredMediaRequired({ s3Enabled: storage.enabled })) { + die( + 'escrita em produção/remoto exige mídia espelhada: configure S3_BUCKET, S3_ENDPOINT, S3_ACCESS_KEY_ID e S3_SECRET_ACCESS_KEY; sem elas o original não está acessível no alvo.', + ) + } + } + + const config = (await import('../src/payload.config.ts')).default + const { listArchivePhotos } = await import('../src/utilities/flickr/archivePhotoIngest.ts') + const { generateArchivePhotoGrade } = + await import('../src/utilities/archivePhotos/archivePhotoThumbnails.ts') + const { privateMediaObjectExists, resolvePrivateMediaStaticDir } = + await import('../src/utilities/privateMedia/privateMediaResponse.ts') + const { ARCHIVE_PHOTO_SLUG } = await import('../src/lib/archivePhoto.ts') + const { archivePhotoGradeFilename } = await import('../src/lib/archivePhotoThumbnail.ts') + + const payload = await getPayload({ config }) + const startedAt = Date.now() + const staticDir = resolvePrivateMediaStaticDir(payload, ARCHIVE_PHOTO_SLUG) + + const rows = await listArchivePhotos(payload) + const eligible = archiveThumbnailEligibleRows(rows) + const scoped = options.only ? eligible.filter((row) => options.only.includes(row.id)) : eligible + const selected = options.limit ? scoped.slice(0, options.limit) : scoped + + console.log( + `[archive:thumbnails] álbum aprovado: ${eligible.length} foto(s) elegível(is); processando ${selected.length}`, + ) + + const gradeExists = (row) => + privateMediaObjectExists({ + filename: archivePhotoGradeFilename(row.filename), + staticDir, + }) + + if (mode === 'plan' || mode === 'verify') { + const probeResults = await mapWithConcurrency(selected, options.concurrency, async (row) => ({ + id: row.id, + flickrId: row.flickrId, + filename: row.filename, + present: await gradeExists(row), + })) + const probe = summarizeArchiveThumbnailProbe(probeResults) + const report = { + runAt, + mode, + target: databaseTarget(), + eligibleTotal: eligible.length, + concurrency: options.concurrency, + probe, + results: probeResults, + durationMs: Date.now() - startedAt, + } + const reportPath = await writeReport(options, report, `-${mode}`) + for (const line of formatArchiveThumbnailProbeReport(report)) console.log(line) + console.log(`\n[archive:thumbnails] recibo JSON: ${reportPath}`) + if (mode === 'verify' && probe.missing > 0) { + die(`${probe.missing} miniatura(s) faltando — aceite do C248 não satisfeito.`) + } + process.exit(0) + } + + const results = await mapWithConcurrency(selected, options.concurrency, async (row) => { + const outcome = (await gradeExists(row)) + ? { status: 'skipped', reason: 'already-present' } + : await generateArchivePhotoGrade(payload, { filename: row.filename }) + const detail = + outcome.status === 'skipped' && outcome.detail + ? ` [${outcome.reason}]: ${outcome.detail}` + : outcome.status === 'failed' + ? ` [${outcome.reason}]` + : '' + console.log(`[archive:thumbnails] ${outcome.status}${detail} ${row.flickrId}`) + return { id: row.id, flickrId: row.flickrId, filename: row.filename, outcome } + }) + + const summary = summarizeArchiveThumbnailRun(results) + const report = { + runAt, + mode, + target: databaseTarget(), + eligibleTotal: eligible.length, + concurrency: options.concurrency, + summary, + results: results.map((result) => ({ + id: result.id, + flickrId: result.flickrId, + filename: result.filename, + status: result.outcome.status, + reason: result.outcome.reason ?? null, + detail: result.outcome.detail ?? null, + bytes: result.outcome.bytes ?? null, + })), + skipped: results + .filter((result) => result.outcome.status === 'skipped') + .map((result) => ({ + id: result.id, + filename: result.filename, + reason: result.outcome.reason, + detail: result.outcome.detail ?? null, + })), + failed: results + .filter((result) => result.outcome.status === 'failed') + .map((result) => ({ + id: result.id, + filename: result.filename, + reason: result.outcome.reason, + })), + durationMs: Date.now() - startedAt, + } + const reportPath = await writeReport(options, report) + for (const line of formatArchiveThumbnailRunReport(report)) console.log(line) + console.log(`\n[archive:thumbnails] recibo JSON: ${reportPath}`) + + if (summary.failed > 0) { + die(`${summary.failed} falha(s) — o recibo lista os motivos; reexecutar retenta.`) + } + process.exit(0) +} + +main().catch((error) => { + die(error?.message || String(error)) +}) diff --git a/scripts/check-archive-integrity.mjs b/scripts/check-archive-integrity.mjs index 48f8749d..a93975b9 100644 --- a/scripts/check-archive-integrity.mjs +++ b/scripts/check-archive-integrity.mjs @@ -49,7 +49,6 @@ import { archiveIntegrityReportStamp, formatArchiveIntegrityRepairReport, formatArchiveIntegrityScanReport, - mapWithConcurrency, parseArchiveIntegrityCliArgs, summarizeArchiveIntegrityRepair, summarizeArchiveIntegrityScan, @@ -62,6 +61,7 @@ import { databaseTarget, dieWithLabel, loadCliEnv, + mapWithConcurrency, mirroredMediaRequired, writeRepoFile, } from './lib/cli.mjs' diff --git a/scripts/lib/archiveIntegrityPlan.mjs b/scripts/lib/archiveIntegrityPlan.mjs index 53d6d6a7..76eb56ce 100644 --- a/scripts/lib/archiveIntegrityPlan.mjs +++ b/scripts/lib/archiveIntegrityPlan.mjs @@ -1,10 +1,13 @@ /** * C246 — pure planning/reporting of the photo-archive integrity sweep: the argv - * parser, the bounded-concurrency map, the scan/repair summaries and the human - * lines + JSON receipt. No I/O of its own: the inspector, the Flickr client and - * the Payload writes are injected, so the unit spec drives everything with fakes. + * parser, the scan/repair summaries and the human lines + JSON receipt. No I/O + * of its own: the inspector, the Flickr client and the Payload writes are + * injected, so the unit spec drives everything with fakes. The bounded map that + * used to live here moved to `cli.mjs` when the C248 backfill became its second + * consumer. */ +import { parseIdList } from './cli.mjs' import { formatArchiveBytes } from './flickrPlan.mjs' const USAGE = @@ -22,26 +25,7 @@ export const archiveIntegrityReportStamp = (runAt) => String(runAt).replace(/[:. * @param {string} value * @returns {number[]} */ -export const parseArchiveIntegrityOnly = (value) => { - const text = String(value ?? '').trim() - if (text === '') - throw new Error(`--only vazio — informe ao menos um id (ex.: --only 80,83) — ${USAGE}.`) - const ids = [] - for (const part of text.split(',')) { - const item = part.trim() - if (!/^\d+$/.test(item)) { - throw new Error( - `--only espera ids numéricos separados por vírgula (recebi "${value}") — ${USAGE}.`, - ) - } - const id = Number(item) - if (!Number.isSafeInteger(id) || id < 1) { - throw new Error(`--only recebeu um id fora do intervalo: "${item}" — ${USAGE}.`) - } - if (!ids.includes(id)) ids.push(id) - } - return ids -} +export const parseArchiveIntegrityOnly = (value) => parseIdList({ value, usage: USAGE }) /** * `pnpm archive:integrity` argv parser — pure, so the unit spec pins it (the @@ -92,36 +76,6 @@ export const parseArchiveIntegrityCliArgs = (argv = process.argv.slice(2)) => { return options } -/** - * Bounded-concurrency map that preserves input order. The sweep downloads and - * decodes 6k+ photos, so unbounded `Promise.all` would be reckless; the pool - * keeps at most `concurrency` workers alive and returns the results in the - * same order as the input (receipts stay deterministic). - * - * @template T, R - * @param {readonly T[]} items - * @param {number} concurrency - * @param {(item: T, index: number) => Promise} worker - * @returns {Promise} - */ -export const mapWithConcurrency = async (items, concurrency, worker) => { - const results = new Array(items.length) - const size = Math.max(1, Math.min(Math.trunc(Number(concurrency)) || 1, items.length)) - let next = 0 - - const runner = async () => { - for (;;) { - const index = next - next += 1 - if (index >= items.length) return - results[index] = await worker(items[index], index) - } - } - - await Promise.all(Array.from({ length: size }, runner)) - return results -} - /** * Aggregates one sweep: what was read whole, what is missing/corrupt, and the * rows kept out of the repair (`removed` is a takedown, C242). `pending` counts diff --git a/scripts/lib/archiveThumbnailPlan.mjs b/scripts/lib/archiveThumbnailPlan.mjs new file mode 100644 index 00000000..3a5bd8a3 --- /dev/null +++ b/scripts/lib/archiveThumbnailPlan.mjs @@ -0,0 +1,208 @@ +/** + * C248 — pure planning/reporting of the stored-grade thumbnail backfill: the + * argv parser, the eligible-row filter, the probe/run summaries and the human + * lines + JSON receipt. No I/O of its own: the object probe, the generation + * and the Payload reads are injected by the CLI, so the unit spec drives + * everything with fakes. + */ + +import { parseIdList } from './cli.mjs' +import { formatArchiveBytes } from './flickrPlan.mjs' + +const USAGE = + 'uso: `pnpm archive:thumbnails [--apply | --verify] [--only ] [--limit ] [--concurrency ] [--out ]`' + +export const ARCHIVE_THUMBNAIL_DEFAULT_OUT_DIR = 'data/archive' + +/** + * The generation is bounded per process by the same 2-slot gate the web heal + * uses (`MAX_CONCURRENT_GENERATIONS`), so a CLI concurrency above 2 would not + * add parallel encodes; the default matches the real ceiling. + */ +export const ARCHIVE_THUMBNAIL_DEFAULT_CONCURRENCY = 2 + +export const archiveThumbnailReportStamp = (runAt) => String(runAt).replace(/[:.]/g, '-') + +/** + * `--only 80,83` → `[80, 83]` (photo ids, the row primary key). Exact digits + * only: a typo must refuse the run, never silently widen or empty the selection. + * + * @param {string} value + * @returns {number[]} + */ +export const parseArchiveThumbnailOnly = (value) => parseIdList({ value, usage: USAGE }) + +/** + * `pnpm archive:thumbnails` argv parser — pure, so the unit spec pins it (the + * CLI owns the exit and the help text). The default mode is the read-only + * plan; `--apply` generates the missing grades and `--verify` checks the + * coverage of the eligible rows. Errors carry the usage line. + * + * @param {string[]} [argv] + * @returns {{ apply: boolean, verify: boolean, only: number[] | null, limit: number | null, concurrency: number, out: string, help: boolean }} + */ +export const parseArchiveThumbnailCliArgs = (argv = process.argv.slice(2)) => { + const options = { + apply: false, + verify: false, + only: null, + limit: null, + concurrency: ARCHIVE_THUMBNAIL_DEFAULT_CONCURRENCY, + out: ARCHIVE_THUMBNAIL_DEFAULT_OUT_DIR, + help: false, + } + + for (let index = 0; index < argv.length; index += 1) { + const arg = argv[index] + const value = () => { + const next = argv[++index] + if (next === undefined || next.startsWith('--')) { + throw new Error(`faltou valor para ${arg} — ${USAGE}.`) + } + return next + } + if (arg === '--help' || arg === '-h') options.help = true + else if (arg === '--apply') options.apply = true + else if (arg === '--verify') options.verify = true + else if (arg === '--only') options.only = parseArchiveThumbnailOnly(value()) + else if (arg === '--limit') options.limit = Number(value()) + else if (arg === '--concurrency') options.concurrency = Number(value()) + else if (arg === '--out') options.out = value() + else throw new Error(`argumento desconhecido: ${arg} — ${USAGE}.`) + } + + if (options.help) return options + if (options.apply && options.verify) { + throw new Error(`--apply e --verify são mutuamente exclusivos — ${USAGE}.`) + } + if (options.limit !== null && (!Number.isInteger(options.limit) || options.limit < 1)) { + throw new Error(`--limit deve ser >= 1 — ${USAGE}.`) + } + if (!Number.isInteger(options.concurrency) || options.concurrency < 1) { + throw new Error(`--concurrency deve ser >= 1 — ${USAGE}.`) + } + if (options.out.split(/[\\/]/).includes('..')) { + throw new Error('--out não pode escapar do diretório do repo (sem "..").') + } + return options +} + +/** + * The eligible queue: an APPROVED photo with a stored filename has a public + * preview to serve, so it deserves a grade. Draft and removed rows stay out + * (the removal guardrail: generating for them would be discarded work), and a + * row with no file can never produce one. + * + * @param {Array<{ id: number, flickrId: string, filename: string | null | undefined, publicationStatus: string }>} rows + * @returns {Array<{ id: number, flickrId: string, filename: string }>} + */ +export const archiveThumbnailEligibleRows = (rows) => + (Array.isArray(rows) ? rows : []) + .filter((row) => row.publicationStatus === 'approved' && Boolean(row.filename)) + .map((row) => ({ id: row.id, flickrId: row.flickrId, filename: row.filename })) + +/** + * Aggregates one read-only probe (plan/verify): how many eligible photos have + * the stored grade and how many still miss it. The probe never throws by + * design (an unreachable store reads as missing), so a completely missing + * coverage in the receipt is also the honest signal to check the bucket. + * + * @param {Array<{ id: number, flickrId: string, present: boolean }>} results + */ +export const summarizeArchiveThumbnailProbe = (results) => { + let present = 0 + let missing = 0 + const missingIds = [] + for (const result of results) { + if (result.present) present += 1 + else { + missing += 1 + missingIds.push(result.id) + } + } + return { eligible: results.length, present, missing, missingIds } +} + +/** + * Aggregates one `--apply` run: what was generated, what was honestly skipped + * (missing/corrupt origin, already present) and what failed and deserves a + * retry. `bytes` sums only the fresh encodes. + * + * @param {Array<{ id: number, outcome: { status: string, reason?: string, detail?: string, bytes?: number } }>} results + */ +export const summarizeArchiveThumbnailRun = (results) => { + let generated = 0 + let skipped = 0 + let failed = 0 + let bytes = 0 + const skippedByReason = {} + for (const result of results) { + const outcome = result.outcome ?? {} + if (outcome.status === 'generated') { + generated += 1 + bytes += Number(outcome.bytes) || 0 + } else if (outcome.status === 'skipped') { + skipped += 1 + const reason = outcome.reason ?? 'unknown' + skippedByReason[reason] = (skippedByReason[reason] ?? 0) + 1 + } else { + // Unknown outcomes are failures: a new status must never hide in "skips". + failed += 1 + } + } + return { eligible: results.length, generated, skipped, failed, bytes, skippedByReason } +} + +const SKIP_LABEL = { + 'already-present': 'já existia', + 'missing-origin': 'original ausente', + 'corrupt-origin': 'original ilegível', + 'no-filename': 'sem arquivo', + 'recent-failure': 'falha recente', +} + +/** + * Human lines of the plan/verify probe; the JSON keeps the full report. + * + * @param {Record} report + * @returns {string[]} + */ +export const formatArchiveThumbnailProbeReport = (report) => { + const label = '[archive:thumbnails]' + const probe = report.probe + return [ + `${label} álbum aprovado: ${probe.eligible} foto(s) · miniatura presente: ${probe.present} · faltando: ${probe.missing}`, + ...(probe.missing > 0 + ? [ + `${label} ${report.mode === 'plan' ? 'rode `pnpm archive:thumbnails --apply` para gerar as faltantes' : 'cobertura incompleta — rode o --apply'}.`, + ] + : []), + ] +} + +/** + * Human lines of the `--apply` receipt, listing every skip and failure with + * its reason. + * + * @param {Record} report + * @returns {string[]} + */ +export const formatArchiveThumbnailRunReport = (report) => { + const label = '[archive:thumbnails]' + const summary = report.summary + const lines = [ + `${label} lote: ${summary.eligible} foto(s) elegível(is) · geradas: ${summary.generated} · puladas: ${summary.skipped} · falhas: ${summary.failed} (${formatArchiveBytes(summary.bytes)})`, + ] + for (const [reason, count] of Object.entries(summary.skippedByReason)) { + lines.push(` - ${SKIP_LABEL[reason] ?? reason}: ${count}`) + } + for (const item of report.skipped ?? []) { + lines.push( + ` - pulada ${item.id} (${item.filename}) [${SKIP_LABEL[item.reason] ?? item.reason}]${item.detail ? `: ${item.detail}` : ''}`, + ) + } + for (const item of report.failed ?? []) { + lines.push(` - falha ${item.id} (${item.filename}): ${item.reason ?? 'erro'}`) + } + return lines +} diff --git a/scripts/lib/cli.mjs b/scripts/lib/cli.mjs index e1081d83..3a9acaf6 100644 --- a/scripts/lib/cli.mjs +++ b/scripts/lib/cli.mjs @@ -486,3 +486,68 @@ export const writeRepoFile = async ({ label, root, relativePath, body }) => { await writeFile(path, body) console.log(`[${label}] wrote ${relativePath} (${Buffer.byteLength(body)} bytes)`) } + +/** + * `--only 80,83` → `[80, 83]` (numeric row ids). Exact digits only: a typo + * must refuse the run, never silently widen or empty the selection. The two + * archive plan-libs (C246/C248) share this one copy; the caller passes its + * flag and usage line so the error message stays domain-specific. + * + * @param {{ value: string, flag?: string, usage?: string }} options + * @returns {number[]} + */ +export const parseIdList = ({ value, flag = '--only', usage = '' }) => { + const text = String(value ?? '').trim() + const suffix = usage ? ` — ${usage}` : '' + if (text === '') { + throw new Error(`${flag} vazio — informe ao menos um id (ex.: ${flag} 80,83)${suffix}.`) + } + const ids = [] + for (const part of text.split(',')) { + const item = part.trim() + if (!/^\d+$/.test(item)) { + throw new Error( + `${flag} espera ids numéricos separados por vírgula (recebi "${value}")${suffix}.`, + ) + } + const id = Number(item) + if (!Number.isSafeInteger(id) || id < 1) { + throw new Error(`${flag} recebeu um id fora do intervalo: "${item}"${suffix}.`) + } + if (!ids.includes(id)) ids.push(id) + } + return ids +} + +/** + * Bounded-concurrency map that preserves input order (C248 — moved here from + * `archiveIntegrityPlan.mjs` on the debt's trigger: the second consumer landed; + * the C246 sweep and the C248 thumbnail backfill share this one copy). The + * sweeps download and decode thousands of photos, so unbounded `Promise.all` + * would be reckless; the pool keeps at most `concurrency` workers alive and + * returns the results in the same order as the input (receipts stay + * deterministic). + * + * @template T, R + * @param {readonly T[]} items + * @param {number} concurrency + * @param {(item: T, index: number) => Promise} worker + * @returns {Promise} + */ +export const mapWithConcurrency = async (items, concurrency, worker) => { + const results = new Array(items.length) + const size = Math.max(1, Math.min(Math.trunc(Number(concurrency)) || 1, items.length)) + let next = 0 + + const runner = async () => { + for (;;) { + const index = next + next += 1 + if (index >= items.length) return + results[index] = await worker(items[index], index) + } + } + + await Promise.all(Array.from({ length: size }, runner)) + return results +} diff --git a/scripts/lib/test-affected-core.mjs b/scripts/lib/test-affected-core.mjs index e715e9c7..2885b8fe 100644 --- a/scripts/lib/test-affected-core.mjs +++ b/scripts/lib/test-affected-core.mjs @@ -48,6 +48,7 @@ export const SCRIPTS_SPEC_PINNED = [ 'scripts/lib/archiveIntegrityPlan.mjs', 'scripts/lib/archiveIntegrityRepair.mjs', 'scripts/lib/archivePublishPlan.mjs', + 'scripts/lib/archiveThumbnailPlan.mjs', 'scripts/lib/archiveVisionApi.mjs', 'scripts/lib/auto-unblock.mjs', 'scripts/lib/briefingContent.mjs', diff --git a/src/app/(frontend)/fotos/[id]/midia/route.ts b/src/app/(frontend)/fotos/[id]/midia/route.ts index d310f6e9..38a862b9 100644 --- a/src/app/(frontend)/fotos/[id]/midia/route.ts +++ b/src/app/(frontend)/fotos/[id]/midia/route.ts @@ -1,13 +1,19 @@ import config from '@payload-config' -import { NextResponse } from 'next/server' +import { after, NextResponse } from 'next/server' import { getPayload } from 'payload' import { ARCHIVE_PHOTO_SLUG } from '@/lib/archivePhoto' import { ARCHIVE_PHOTO_THUMB_WIDTH } from '@/lib/archivePhotoPublicCatalog' -import { PRIVATE_MEDIA_CACHE_CONTROL } from '@/lib/privateMedia' +import { + ARCHIVE_PHOTO_GRADE_MIME_TYPE, + archivePhotoGradeFilename, +} from '@/lib/archivePhotoThumbnail' +import { acceptsAvif, PRIVATE_MEDIA_CACHE_CONTROL } from '@/lib/privateMedia' import { getApprovedArchivePhotoById } from '@/utilities/archivePhotos/archivePhotoReads' +import { ensureArchivePhotoGrade } from '@/utilities/archivePhotos/archivePhotoThumbnails' import { buildPrivateMediaImageResponse, + buildPrivateMediaResponse, resolvePrivateMediaStaticDir, } from '@/utilities/privateMedia/privateMediaResponse' @@ -17,11 +23,18 @@ export const dynamic = 'force-dynamic' * C233 — the only public door to an approved photo. The gate is the cached * public read (`archivePhotoIsPublic`), so a draft, a removed photo, an unknown * id and a delete all answer the same silent 404 — the route never leaks which - * photos exist. `?tamanho=grade` serves the on-the-fly resized thumbnail, - * `?download=1` forces the legible `jorge-solla-1313-foto-.` name; the - * object itself stays in the private bucket (the C193/C199 streaming helper is - * the single owner of that I/O). `X-Robots-Tag` keeps the derivative out of the - * index: the canonical surface is the album, never the file. + * photos exist. `?tamanho=grade` serves the grade thumbnail, `?download=1` + * forces the legible `jorge-solla-1313-foto-.` name; the object itself + * stays in the private bucket (the C193/C199 streaming helper is the single + * owner of that I/O). `X-Robots-Tag` keeps the derivative out of the index: the + * canonical surface is the album, never the file. + * + * C248 — when the client accepts AVIF and the stored grade exists, the grade + * object is served directly (no sharp on the request); on a miss or any read + * failure the route answers the on-the-fly JPEG exactly as before and heals + * the grade for the next request in `after()`. `Vary: Accept` keeps a heuristic + * cache from mixing the two representations even though the response is + * `no-store`. */ const notFound = (): NextResponse => @@ -52,14 +65,42 @@ export const GET = async ( }) .catch(() => null) if (!media?.filename) return notFound() + const originalFilename = media.filename const searchParams = new URL(request.url).searchParams const thumbnail = searchParams.get('tamanho') === 'grade' const download = searchParams.get('download') === '1' + const staticDir = resolvePrivateMediaStaticDir(payload, ARCHIVE_PHOTO_SLUG) + + if (thumbnail && !download && acceptsAvif(request.headers.get('accept'))) { + // The stored grade is a sibling object in the same private store; the + // streaming owner answers 404/416 without leaking which object is there. + const grade = await buildPrivateMediaResponse({ + media: { + filename: archivePhotoGradeFilename(originalFilename), + mimeType: ARCHIVE_PHOTO_GRADE_MIME_TYPE, + }, + staticDir, + rangeHeader: null, + download: false, + dispositionFilename: photo.downloadFilename, + }).catch(() => null) + + if (grade?.status === 200) { + grade.headers.set('X-Robots-Tag', 'noindex') + grade.headers.set('Vary', 'Accept') + return grade + } + + // Missing/unreadable grade: rebuild it out of this latency window; the + // fallback below answers this request exactly as the pre-C248 route did. + after(() => ensureArchivePhotoGrade(payload, { filename: originalFilename })) + } + const response = await buildPrivateMediaImageResponse({ media, - staticDir: resolvePrivateMediaStaticDir(payload, ARCHIVE_PHOTO_SLUG), + staticDir, width: thumbnail ? ARCHIVE_PHOTO_THUMB_WIDTH : null, download, dispositionFilename: photo.downloadFilename, @@ -69,5 +110,6 @@ export const GET = async ( }) response.headers.set('X-Robots-Tag', 'noindex') + if (thumbnail && !download) response.headers.set('Vary', 'Accept') return response } diff --git a/src/collections/ArchivePhoto.ts b/src/collections/ArchivePhoto.ts index f4cb844e..73c6250e 100644 --- a/src/collections/ArchivePhoto.ts +++ b/src/collections/ArchivePhoto.ts @@ -25,6 +25,7 @@ import { isArchivePhotoRemovalChannelUrl, } from '@/lib/archivePhotoCatalog' import { SPEECH_TOPICS } from '@/lib/speechFacets' +import { ensureArchivePhotoGrade } from '@/utilities/archivePhotos/archivePhotoThumbnails' import { canReadArchivePhoto, payloadAdminOnly } from '@/utilities/campaignAccess' import { revalidateArchivePhotosListing } from '@/utilities/documents' import { purgeFaceDescriptorsForPhoto } from '@/utilities/faceIndex/faceDescriptorIndex' @@ -294,6 +295,26 @@ const revalidateArchivePhotosListingAfterDelete: CollectionAfterDeleteHook = ({ return doc } +/** + * C248 — the approval seam: a photo that ENTERS `approved` gets its stored + * grade warmed best-effort. Fire-and-forget by design — the encode is not part + * of the publication (a failure never blocks the approval) and the route's + * lazy heal covers whatever this loses (a CLI approving in batch exits before + * the promise settles, a restart, a transient store error). Every other write + * (cataloguing, face index, C246 repair) keeps the status, so it never + * triggers a regeneration. + */ +const warmArchivePhotoGradeAfterChange: CollectionAfterChangeHook = ({ doc, previousDoc, req }) => { + if ( + doc.publicationStatus === 'approved' && + previousDoc?.publicationStatus !== 'approved' && + doc.filename + ) { + void ensureArchivePhotoGrade(req.payload, { filename: doc.filename }).catch(() => undefined) + } + return doc +} + /** * C242 — a photo that stops being `approved` cannot keep answering the selfie * search: the anonymous descriptor rows die with the public status (and on @@ -376,7 +397,11 @@ export const ArchivePhoto: CollectionConfig = { }, hooks: { beforeValidate: [deriveArchivePhotoCatalogIndex, requireRemovalChannelForApproval], - afterChange: [revalidateArchivePhotosListingAfterChange, purgeFaceDescriptorsAfterChange], + afterChange: [ + revalidateArchivePhotosListingAfterChange, + purgeFaceDescriptorsAfterChange, + warmArchivePhotoGradeAfterChange, + ], afterDelete: [revalidateArchivePhotosListingAfterDelete, purgeFaceDescriptorsAfterDelete], }, fields: [ diff --git a/src/lib/archivePhotoThumbnail.ts b/src/lib/archivePhotoThumbnail.ts new file mode 100644 index 00000000..874fd238 --- /dev/null +++ b/src/lib/archivePhotoThumbnail.ts @@ -0,0 +1,35 @@ +/** + * C248 — pure rules of the stored grade thumbnail of an archive photo: the + * deterministic sibling filename and the AVIF container/quality of the + * derivative. No I/O and no `server-only`: the public media route, the + * generation utility, the collection hook and the backfill CLI share this + * module, so the naming and the quality live in one place. + * + * The derivative is a SIBLING object of the stored original, never a second + * row: the original keeps its deterministic key (`flickr-.`) and the + * grade is discovered by the exact derived key (`flickr--grade.avif`). + * Nothing is ever listed to find it, so a miss can only mean "not generated + * yet" — the route falls back on the fly and heals. + */ + +/** AVIF at q60 — the repo ruler for web assets (`scripts/lib/imageResize.mjs`). */ +export const ARCHIVE_PHOTO_GRADE_QUALITY = 60 + +/** The stored grade is AVIF; `image/avif` is already inline-safe in the contract. */ +export const ARCHIVE_PHOTO_GRADE_MIME_TYPE = 'image/avif' + +/** The single extension of the derived object. */ +export const ARCHIVE_PHOTO_GRADE_EXTENSION = '.avif' + +/** + * `flickr-123.jpg` → `flickr-123-grade.avif`: the suffix goes BEFORE the + * extension of the original and a name with no usable extension keeps the + * whole name as the base. Deterministic and collision-free with the original + * key, so the same photo always resolves to the same derived key. + */ +export const archivePhotoGradeFilename = (originalFilename: string): string => { + const name = originalFilename.trim() + const dot = name.lastIndexOf('.') + const base = dot > 0 ? name.slice(0, dot) : name + return `${base}-grade${ARCHIVE_PHOTO_GRADE_EXTENSION}` +} diff --git a/src/lib/privateMedia.ts b/src/lib/privateMedia.ts index 4547c7c9..eebb44c8 100644 --- a/src/lib/privateMedia.ts +++ b/src/lib/privateMedia.ts @@ -71,6 +71,33 @@ export const isPrivateMediaMissingError = (error: unknown): boolean => { ) } +/** + * C248 — AVIF negotiation for the stored grade thumbnail. The token must be + * EXPLICIT (`image/avif`) with a positive quality: a missing header, a + * wildcard or `image/jpeg` stays on the JPEG fallback, so every client that + * never opted in (curl, the e2e API request context, a browser without the + * decoder) keeps the exact answer it has today. Pure, so the route and the + * unit spec share the classification. + */ +export const acceptsAvif = (acceptHeader: string | null | undefined): boolean => { + if (typeof acceptHeader !== 'string') return false + for (const part of acceptHeader.split(',')) { + const [rawType, ...params] = part.split(';') + if (rawType.trim().toLowerCase() !== 'image/avif') continue + + let quality = 1 + for (const param of params) { + const [rawKey, rawValue] = param.split('=') + if (rawKey?.trim().toLowerCase() !== 'q') continue + const parsed = Number(rawValue) + // A malformed weight must never arm AVIF: only a real positive q does. + quality = Number.isFinite(parsed) ? parsed : 0 + } + return quality > 0 + } + return false +} + const isInlineSafe = (mimeType: unknown): mimeType is string => typeof mimeType === 'string' && (PRIVATE_MEDIA_INLINE_MIME_TYPES as readonly string[]).includes(mimeType) diff --git a/src/utilities/archivePhotos/archivePhotoThumbnails.ts b/src/utilities/archivePhotos/archivePhotoThumbnails.ts new file mode 100644 index 00000000..4d0efab1 --- /dev/null +++ b/src/utilities/archivePhotos/archivePhotoThumbnails.ts @@ -0,0 +1,222 @@ +import 'server-only' + +import { mkdtemp, rm } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import type { Payload } from 'payload' + +import { ARCHIVE_PHOTO_SLUG } from '@/lib/archivePhoto' +import { ARCHIVE_PHOTO_THUMB_WIDTH } from '@/lib/archivePhotoPublicCatalog' +import { + ARCHIVE_PHOTO_GRADE_MIME_TYPE, + ARCHIVE_PHOTO_GRADE_QUALITY, + archivePhotoGradeFilename, +} from '@/lib/archivePhotoThumbnail' +import { isPrivateMediaMissingError } from '@/lib/privateMedia' +import { messageOf } from '@/utilities/media/ffmpeg' +import { + downloadPrivateMediaToFile, + encodePrivateImageFile, + privateMediaObjectExists, + resolvePrivateMediaStaticDir, + writePrivateObject, +} from '@/utilities/privateMedia/privateMediaResponse' + +/** + * C248 — the stored grade thumbnail of an archive photo: the 720px AVIF q60 + * the public grid serves when it exists. The route tries the deterministic + * sibling key first; on a miss it answers the on-the-fly JPEG and calls + * `ensureArchivePhotoGrade` in `after()`, so the next request hits the stored + * derivative. The collection hook warms the same path when a photo is approved + * and the backfill CLI covers the pre-existing archive. + * + * Nothing here touches the database or the original: the grade is a cache + * entry in the same private store, discovered by exact key. Every failure is + * classified and never thrown — the fallback is the product promise, so a + * broken original or an unreachable store can only mean "skip", never a 500. + */ + +/** Concurrent generations per process: a grid miss can ask for several at once. */ +const MAX_CONCURRENT_GENERATIONS = 2 + +/** A failure is not retried on every view (the molde of C226). */ +const FAILURE_TTL_MS = 5 * 60_000 + +const GENERATION_FAILURE_FALLBACK = 'Falha ao gerar a miniatura da foto.' + +/** What one generation pass can produce (the CLI receipt vocabulary). */ +export type ArchivePhotoGradeGeneration = + | { status: 'generated'; filename: string; bytes: number } + | { + status: 'skipped' + reason: 'no-filename' | 'missing-origin' | 'corrupt-origin' + detail?: string + } + | { status: 'failed'; reason: string } + +/** Add the probe/failure-memo results only `ensureArchivePhotoGrade` produces. */ +export type ArchivePhotoGradeOutcome = + | ArchivePhotoGradeGeneration + | { status: 'skipped'; reason: 'already-present' | 'recent-failure' } + +// Process-wide state (same shape as the C226 frame job): the in-memory slot +// gate keeps one Node server from stampeding sharp, the in-flight map +// deduplicates concurrent requests for the same photo and the failure memo +// stops one bad original from being re-decoded on every view. +let activeGenerations = 0 +const waiting: Array<() => void> = [] +const inFlight = new Map>() +const failures = new Map() + +const rememberFailure = (gradeFilename: string): void => { + const now = Date.now() + for (const [key, at] of failures) { + if (now - at >= FAILURE_TTL_MS) failures.delete(key) + } + failures.set(gradeFilename, now) +} + +const acquireGenerationSlot = async (): Promise<() => void> => { + if (activeGenerations >= MAX_CONCURRENT_GENERATIONS) { + await new Promise((resolve) => waiting.push(resolve)) + } else { + activeGenerations += 1 + } + + let released = false + return () => { + if (released) return + released = true + const next = waiting.shift() + if (next) next() + else activeGenerations -= 1 + } +} + +/** + * One grade, from the stored original to the sibling AVIF. The original is + * downloaded to a caller-owned temp file first: that is what tells a missing + * object (honest skip, the C246 vocabulary) from a corrupt one (decode + * failure, honest skip with the reason in the receipt) from a write failure + * (retryable). Never throws. + */ +export const generateArchivePhotoGrade = async ( + payload: Payload, + { filename }: { filename: string | null | undefined }, +): Promise => { + if (!filename) return { status: 'skipped', reason: 'no-filename' } + + const staticDir = resolvePrivateMediaStaticDir(payload, ARCHIVE_PHOTO_SLUG) + const gradeFilename = archivePhotoGradeFilename(filename) + + const release = await acquireGenerationSlot() + let tempDir: string | null = null + try { + tempDir = await mkdtemp(join(tmpdir(), 'archive-photo-grade-')) + const sourcePath = join(tempDir, 'source') + + try { + await downloadPrivateMediaToFile({ + media: { filename }, + staticDir, + destinationPath: sourcePath, + }) + } catch (error) { + if (isPrivateMediaMissingError(error)) { + return { status: 'skipped', reason: 'missing-origin' } + } + return { + status: 'failed', + reason: `download: ${messageOf(error, GENERATION_FAILURE_FALLBACK)}`, + } + } + + let image: Buffer + try { + image = await encodePrivateImageFile({ + inputPath: sourcePath, + width: ARCHIVE_PHOTO_THUMB_WIDTH, + format: 'avif', + quality: ARCHIVE_PHOTO_GRADE_QUALITY, + }) + } catch (error) { + return { + status: 'skipped', + reason: 'corrupt-origin', + detail: messageOf(error, GENERATION_FAILURE_FALLBACK), + } + } + + try { + await writePrivateObject({ + filename: gradeFilename, + staticDir, + data: image, + mimeType: ARCHIVE_PHOTO_GRADE_MIME_TYPE, + }) + } catch (error) { + return { + status: 'failed', + reason: `upload: ${messageOf(error, GENERATION_FAILURE_FALLBACK)}`, + } + } + + return { status: 'generated', filename: gradeFilename, bytes: image.length } + } catch (error) { + return { status: 'failed', reason: messageOf(error, GENERATION_FAILURE_FALLBACK) } + } finally { + if (tempDir) { + await rm(tempDir, { recursive: true, force: true }).catch(() => undefined) + } + release() + } +} + +/** + * Returns the stored grade of a photo, generating it on a miss. The probe and + * the generation share the in-flight map, so a grid that asks for the same + * missing photo twice runs sharp once. Never throws: a failure degrades to the + * route's on-the-fly fallback and is memoized for a short while. + */ +export const ensureArchivePhotoGrade = async ( + payload: Payload, + { filename }: { filename: string | null | undefined }, +): Promise => { + if (!filename) return { status: 'skipped', reason: 'no-filename' } + + const gradeFilename = archivePhotoGradeFilename(filename) + + const failedAt = failures.get(gradeFilename) + if (failedAt !== undefined && Date.now() - failedAt < FAILURE_TTL_MS) { + return { status: 'skipped', reason: 'recent-failure' } + } + + const staticDir = resolvePrivateMediaStaticDir(payload, ARCHIVE_PHOTO_SLUG) + const exists = await privateMediaObjectExists({ filename: gradeFilename, staticDir }).catch( + () => false, + ) + if (exists) return { status: 'skipped', reason: 'already-present' } + + const running = inFlight.get(gradeFilename) + if (running) return running + + const generation = generateArchivePhotoGrade(payload, { filename }) + .then((result) => { + if (result.status !== 'generated') rememberFailure(gradeFilename) + return result + }) + .catch((error: unknown) => { + rememberFailure(gradeFilename) + console.warn( + `[archive-photo-grade] ${gradeFilename}: ${messageOf(error, GENERATION_FAILURE_FALLBACK)}`, + ) + const reason = messageOf(error, GENERATION_FAILURE_FALLBACK) + return { status: 'failed' as const, reason } + }) + .finally(() => { + inFlight.delete(gradeFilename) + }) + + inFlight.set(gradeFilename, generation) + return generation +} diff --git a/src/utilities/privateMedia/privateMediaResponse.ts b/src/utilities/privateMedia/privateMediaResponse.ts index 4bb45d60..c475bd7b 100644 --- a/src/utilities/privateMedia/privateMediaResponse.ts +++ b/src/utilities/privateMedia/privateMediaResponse.ts @@ -2,9 +2,9 @@ import 'server-only' import type { Config } from '@/payload-types' -import { GetObjectCommand, HeadObjectCommand, S3Client } from '@aws-sdk/client-s3' +import { GetObjectCommand, HeadObjectCommand, PutObjectCommand, S3Client } from '@aws-sdk/client-s3' import { createReadStream, createWriteStream } from 'fs' -import { stat } from 'fs/promises' +import { mkdir, rename, rm, stat, writeFile } from 'fs/promises' import path from 'path' import type { Payload } from 'payload' import { getRangeRequestInfo } from 'payload/internal' @@ -170,6 +170,77 @@ const openPrivateObject = async ({ : openLocalObject({ staticDir, filename, rangeHeader }) } +/** + * C248 — existence probe of one private object in whichever store is + * configured (S3 HEAD or local `stat`). Never throws: an unreachable store + * reads as "not there", so the caller falls back to the on-the-fly path and + * the generation is retried later; the worst case is redundant, idempotent + * work, never a broken response. + */ +export const privateMediaObjectExists = async ({ + filename, + staticDir, +}: { + filename: string + staticDir: string +}): Promise => { + try { + const storage = privateMediaStorage() + if (storage) { + await storage.client.send(new HeadObjectCommand({ Bucket: storage.bucket, Key: filename })) + return true + } + + await stat(resolveLocalPath(staticDir, filename)) + return true + } catch { + return false + } +} + +/** + * C248 — writes one derived object to whichever store is configured. On the + * local disk the bytes land through a same-directory temp file + rename, so a + * crash never leaves a half-written artifact at the canonical key; on S3 the + * PUT is atomic by contract. The key and the MIME ride explicitly — there is + * no upload row behind a derived object. + */ +export const writePrivateObject = async ({ + filename, + staticDir, + data, + mimeType, +}: { + filename: string + staticDir: string + data: Buffer + mimeType: string +}): Promise => { + const storage = privateMediaStorage() + if (storage) { + await storage.client.send( + new PutObjectCommand({ + Bucket: storage.bucket, + Key: filename, + Body: data, + ContentType: mimeType, + }), + ) + return + } + + const filePath = resolveLocalPath(staticDir, filename) + await mkdir(path.dirname(filePath), { recursive: true }) + const tempPath = `${filePath}.${process.pid}.${Date.now()}.tmp` + try { + await writeFile(tempPath, data) + await rename(tempPath, filePath) + } catch (error) { + await rm(tempPath, { force: true }).catch(() => undefined) + throw error + } +} + const notFound = (): Response => new Response(null, { status: 404, headers: { 'Cache-Control': PRIVATE_MEDIA_CACHE_CONTROL } }) @@ -219,6 +290,86 @@ export const buildPrivateMediaResponse = async ({ } } +export type PrivateMediaImageFormat = 'jpeg' | 'avif' + +/** The resize the public grid has always served on the fly (C233). */ +const FALLBACK_JPEG_QUALITY = 80 + +/** + * The shared sharp chain: `failOn: 'error'` keeps a truncated source from + * becoming a half-decoded image and `rotate()` honors the EXIF orientation the + * grid always did. `source` is the local input path, or `undefined` for the + * stream path (the caller pipes into the returned transformer). + */ +const imageEncoder = ( + source: string | undefined, + { width, format, quality }: { width: number; format: PrivateMediaImageFormat; quality: number }, +): sharp.Sharp => { + // sharp rejects `sharp(undefined, options)`; the stream path omits the input + // entirely (the caller pipes into the returned transformer). + const transformer = (source ? sharp(source, { failOn: 'error' }) : sharp({ failOn: 'error' })) + .rotate() + .resize({ width, withoutEnlargement: true }) + return format === 'avif' ? transformer.avif({ quality }) : transformer.jpeg({ quality }) +} + +/** + * C248 — encodes one already-downloaded local image (the stored grade + * generation). The temp file the caller owns is what tells a missing object + * from a corrupt one, so this side is only the decode+encode. + */ +export const encodePrivateImageFile = async ({ + inputPath, + width, + format, + quality, +}: { + inputPath: string + width: number + format: PrivateMediaImageFormat + quality: number +}): Promise => imageEncoder(inputPath, { width, format, quality }).toBuffer() + +/** + * C233/C248 — encodes one image straight from the private store (the public + * grid's on-the-fly path). The stream never becomes a temp file: sharp's + * constructor takes no stream, so the documented path is piping into the + * instance and draining its output. + */ +export const encodePrivateMediaImage = async ({ + media, + staticDir, + width, + format, + quality, +}: { + media: PrivateMediaFile + staticDir: string + width: number + format: PrivateMediaImageFormat + quality: number +}): Promise => { + const filename = media.filename + if (!filename) throw new Error('Mídia privada sem nome de arquivo.') + + const opened = await openPrivateObject({ filename, staticDir, rangeHeader: null }) + if (!(opened.body instanceof Readable)) { + throw new Error(`Objeto privado sem stream: ${filename}`) + } + + const encoded = imageEncoder(undefined, { width, format, quality }) + const source = opened.body + const resized = new Promise((resolve, reject) => { + const chunks: Buffer[] = [] + encoded.on('data', (chunk: Buffer) => chunks.push(chunk)) + encoded.on('end', () => resolve(Buffer.concat(chunks))) + encoded.on('error', reject) + source.on('error', reject) + }) + source.pipe(encoded) + return resized +} + /** * C233 — one image artifact, optionally resized for a public grid. With no * `width` it delegates to `buildPrivateMediaResponse` (range, stored MIME, @@ -256,27 +407,13 @@ export const buildPrivateMediaImageResponse = async ({ if (!filename) return notFound() try { - const opened = await openPrivateObject({ filename, staticDir, rangeHeader: null }) - if (!(opened.body instanceof Readable)) { - throw new Error(`Objeto privado sem stream: ${filename}`) - } - - // sharp's constructor takes no stream; the documented stream path is - // piping into the instance and draining its output. - const transformer = sharp({ failOn: 'error' }) - .rotate() - .resize({ width, withoutEnlargement: true }) - .jpeg({ quality: 80 }) - const source = opened.body - const resized = new Promise((resolve, reject) => { - const chunks: Buffer[] = [] - transformer.on('data', (chunk: Buffer) => chunks.push(chunk)) - transformer.on('end', () => resolve(Buffer.concat(chunks))) - transformer.on('error', reject) - source.on('error', reject) + const image = await encodePrivateMediaImage({ + media, + staticDir, + width, + format: 'jpeg', + quality: FALLBACK_JPEG_QUALITY, }) - source.pipe(transformer) - const image = await resized return new Response(new Uint8Array(image), { status: 200, diff --git a/tests/e2e/frontendFotos.e2e.spec.ts b/tests/e2e/frontendFotos.e2e.spec.ts index 30615df2..b3d5f067 100644 --- a/tests/e2e/frontendFotos.e2e.spec.ts +++ b/tests/e2e/frontendFotos.e2e.spec.ts @@ -356,7 +356,12 @@ test.describe('Frontend Álbum de Fotos (C233)', () => { `${BASE_URL}/fotos/${approvedFeira}/midia?tamanho=grade`, ) expect(thumbnailResponse.status()).toBe(200) + // The API request context sends `Accept: */*` — never an explicit + // `image/avif` — so the JPEG fallback is the deterministic answer here + // (C248); `Vary: Accept` is the negotiation contract pinned. expect(thumbnailResponse.headers()['content-type']).toBe('image/jpeg') + // Next merges its own RSC/router tokens into `Vary`; ours must be there. + expect(thumbnailResponse.headers()['vary']).toContain('Accept') expect(thumbnailResponse.headers()['x-robots-tag']).toBe('noindex') expect(thumbnailResponse.headers()['cache-control']).toContain('no-store') diff --git a/tests/int/archivePhotoPublicRead.int.spec.ts b/tests/int/archivePhotoPublicRead.int.spec.ts index 4ac1d73c..7fad230f 100644 --- a/tests/int/archivePhotoPublicRead.int.spec.ts +++ b/tests/int/archivePhotoPublicRead.int.spec.ts @@ -14,8 +14,16 @@ vi.mock('next/cache', () => ({ unstable_cache: (fn: unknown) => fn, })) +vi.mock('next/server', async (importOriginal) => { + const actual = await importOriginal() + // C248 — the route heals the grade with `after()`, which requires a request + // scope; the generation itself is exercised in its own spec. + return { ...actual, after: () => undefined } +}) + import { GET as getArchivePhotoMedia } from '@/app/(frontend)/fotos/[id]/midia/route' import { ARCHIVE_PHOTO_SLUG, type ArchivePhotoImport } from '@/lib/archivePhoto' +import { archivePhotoGradeFilename } from '@/lib/archivePhotoThumbnail' import config from '@/payload.config' import { getApprovedArchivePhotoById, @@ -23,6 +31,7 @@ import { hasPublishedArchivePhotos, } from '@/utilities/archivePhotos/archivePhotoReads' import { ingestArchivePhoto } from '@/utilities/flickr/archivePhotoIngest' +import { resolvePrivateMediaStaticDir } from '@/utilities/privateMedia/privateMediaResponse' import { ARCHIVE_PHOTO_JPEG_BYTES } from '../helpers/archivePhotoFixture' import { installCampaignFixtures } from '../helpers/campaignFixtures' @@ -129,7 +138,23 @@ describe('public photo album reads (C233)', () => { }) afterAll(async () => { + const staticDir = resolvePrivateMediaStaticDir(payload, ARCHIVE_PHOTO_SLUG) for (const flickrId of createdFlickrIds) { + // The approval hook warms the C248 grade as a sibling object; the delete + // below removes the original, so the derived file is cleaned here. + const photo = await payload.find({ + collection: ARCHIVE_PHOTO_SLUG, + where: { flickrId: { equals: flickrId } }, + depth: 0, + limit: 1, + overrideAccess: true, + }) + const filename = photo.docs[0]?.filename + if (filename) { + await rm(join(staticDir, archivePhotoGradeFilename(filename)), { force: true }).catch( + () => undefined, + ) + } await payload.delete({ collection: ARCHIVE_PHOTO_SLUG, where: { flickrId: { equals: flickrId } }, @@ -302,6 +327,7 @@ describe('public photo album reads (C233)', () => { expect(thumbnail.status).toBe(200) expect(thumbnail.headers.get('Content-Type')).toBe('image/jpeg') expect(thumbnail.headers.get('X-Robots-Tag')).toBe('noindex') + expect(thumbnail.headers.get('Vary')).toBe('Accept') const download = await getArchivePhotoMedia( new Request(`http://localhost/fotos/${approved.id}/midia?download=1`), diff --git a/tests/int/archivePhotoThumbnails.int.spec.ts b/tests/int/archivePhotoThumbnails.int.spec.ts new file mode 100644 index 00000000..bc3577b9 --- /dev/null +++ b/tests/int/archivePhotoThumbnails.int.spec.ts @@ -0,0 +1,277 @@ +// @vitest-environment node + +import { randomUUID } from 'node:crypto' +import { mkdtemp, readFile, rm, stat, writeFile } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import type { Payload } from 'payload' +import { getPayload } from 'payload' +import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from 'vitest' + +const { afterMock } = vi.hoisted(() => ({ afterMock: vi.fn() })) + +vi.mock('next/cache', () => ({ + revalidatePath: () => undefined, + revalidateTag: vi.fn(), + unstable_cache: (fn: unknown) => fn, +})) + +vi.mock('next/server', async (importOriginal) => { + const actual = await importOriginal() + // The route heals the grade with `after()`, which requires a request scope; + // the generation itself is exercised directly below. The call is captured + // here so the wiring (and its absence on a hit) is pinned. + return { ...actual, after: afterMock } +}) + +import { GET as getArchivePhotoMedia } from '@/app/(frontend)/fotos/[id]/midia/route' +import { ARCHIVE_PHOTO_SLUG, type ArchivePhotoImport } from '@/lib/archivePhoto' +import { archivePhotoGradeFilename } from '@/lib/archivePhotoThumbnail' +import config from '@/payload.config' +import { + ensureArchivePhotoGrade, + generateArchivePhotoGrade, +} from '@/utilities/archivePhotos/archivePhotoThumbnails' +import { ingestArchivePhoto } from '@/utilities/flickr/archivePhotoIngest' +import { resolvePrivateMediaStaticDir } from '@/utilities/privateMedia/privateMediaResponse' + +import { ARCHIVE_PHOTO_JPEG_BYTES } from '../helpers/archivePhotoFixture' +import { installCampaignFixtures } from '../helpers/campaignFixtures' +import { PHOTO_ALBUM_LEASE_KEY, withExclusiveTestDatabaseLease } from '../helpers/testDatabaseLease' + +// C248 — the stored grade thumbnail over the real Payload `teqo_test` and the +// real local storage: generation/idempotence/failure classification, the +// sibling-key contract and the route matrix (stored AVIF on `Accept`, JPEG +// fallback otherwise, miss heals in the background). + +let payload: Payload +const createdFlickrIds = new Set() +const tempDirs: string[] = [] + +installCampaignFixtures({ + getPayload: () => payload, + setPayload: (nextPayload) => { + payload = nextPayload + }, +}) + +const CHANNEL = 'https://jorgessolla.example.org/remocao' + +const record = ( + flickrId: string, + overrides: Partial = {}, +): ArchivePhotoImport => ({ + flickrId, + sourceUrl: `https://www.flickr.com/photos/depjorgesolla/${flickrId}/`, + owner: '12345678@N00', + license: '0', + title: `Foto ${flickrId}`, + description: null, + tags: [], + takenAt: '2025-03-14 10:20:30', + postedAt: '2025-03-15T10:00:00.000Z', + albums: [], + geo: null, + exif: [], + originalUrl: `https://live.staticflickr.com/65535/${flickrId}_abcdef_o.jpg`, + originalKind: 'original', + ...overrides, +}) + +const createPhoto = async (): Promise<{ id: number; flickrId: string; filename: string }> => { + const flickrId = `c248-${randomUUID().slice(0, 8)}` + createdFlickrIds.add(flickrId) + const dir = await mkdtemp(join(tmpdir(), 'archive-photo-thumb-')) + tempDirs.push(dir) + const filePath = join(dir, `flickr-${flickrId}.jpg`) + await writeFile(filePath, ARCHIVE_PHOTO_JPEG_BYTES) + const created = await ingestArchivePhoto(payload, record(flickrId), { filePath }) + if (created.status !== 'created' || !created.filename) { + throw new Error(`fixture not created: ${created.status}`) + } + return { id: created.id, flickrId, filename: created.filename } +} + +const setChannel = async (removalChannelUrl: string): Promise => { + await withExclusiveTestDatabaseLease(payload, PHOTO_ALBUM_LEASE_KEY, () => + payload.updateGlobal({ + slug: 'photoAlbum', + data: { published: true, removalChannelUrl }, + overrideAccess: true, + }), + ) +} + +const closeAlbumAndClearChannel = async (): Promise => { + await withExclusiveTestDatabaseLease(payload, PHOTO_ALBUM_LEASE_KEY, () => + payload.updateGlobal({ + slug: 'photoAlbum', + data: { published: false, removalChannelUrl: null }, + overrideAccess: true, + }), + ) +} + +const approve = async (id: number): Promise => { + await payload.update({ + collection: ARCHIVE_PHOTO_SLUG, + id, + data: { publicationStatus: 'approved' }, + overrideAccess: true, + }) +} + +const staticDir = (): string => resolvePrivateMediaStaticDir(payload, ARCHIVE_PHOTO_SLUG) + +const gradePathOf = (filename: string): string => + join(staticDir(), archivePhotoGradeFilename(filename)) + +const exists = async (path: string): Promise => + stat(path).then( + () => true, + () => false, + ) + +// The approval hook warms the grade fire-and-forget; awaiting `ensure` lands on +// the same in-flight generation, so the file is settled when it resolves. +const settleGrade = async (filename: string): Promise => { + await ensureArchivePhotoGrade(payload, { filename }) +} + +describe('archive photo grade thumbnails (C248)', () => { + beforeAll(async () => { + payload = await getPayload({ config: await config }) + await setChannel(CHANNEL) + }) + + beforeEach(() => { + afterMock.mockClear() + }) + + afterAll(async () => { + for (const flickrId of createdFlickrIds) { + const photo = await payload.find({ + collection: ARCHIVE_PHOTO_SLUG, + where: { flickrId: { equals: flickrId } }, + depth: 0, + limit: 1, + overrideAccess: true, + }) + const filename = photo.docs[0]?.filename + if (filename) { + await rm(gradePathOf(filename), { force: true }).catch(() => undefined) + } + await payload.delete({ + collection: ARCHIVE_PHOTO_SLUG, + where: { flickrId: { equals: flickrId } }, + overrideAccess: true, + }) + } + await closeAlbumAndClearChannel() + await Promise.all(tempDirs.map((dir) => rm(dir, { recursive: true, force: true }))) + }) + + it('generates the sibling AVIF once and converges on re-run', async () => { + const photo = await createPhoto() + const gradePath = gradePathOf(photo.filename) + await rm(gradePath, { force: true }) + + const generated = await generateArchivePhotoGrade(payload, { filename: photo.filename }) + expect(generated.status).toBe('generated') + if (generated.status === 'generated') { + expect(generated.filename).toBe(archivePhotoGradeFilename(photo.filename)) + expect(generated.bytes).toBeGreaterThan(0) + } + + // ISO-BMFF container with the AVIF brand: a real AVIF came out of sharp. + const bytes = await stat(gradePath) + expect(bytes.size).toBeGreaterThan(0) + const file = await readFile(gradePath) + expect(file.subarray(4, 8).toString('latin1')).toBe('ftyp') + expect(file.subarray(8, 12).toString('latin1')).toContain('avi') + + // The existing grade is never regenerated; the ensure probe converges. + expect(await ensureArchivePhotoGrade(payload, { filename: photo.filename })).toMatchObject({ + status: 'skipped', + reason: 'already-present', + }) + }) + + it('warms the grade when the photo enters approval', async () => { + const photo = await createPhoto() + await rm(gradePathOf(photo.filename), { force: true }) + + await approve(photo.id) + await settleGrade(photo.filename) + + expect(await exists(gradePathOf(photo.filename))).toBe(true) + }) + + it('classifies a missing original as an honest skip', async () => { + const photo = await createPhoto() + await rm(join(staticDir(), photo.filename), { force: true }) + + expect(await generateArchivePhotoGrade(payload, { filename: photo.filename })).toMatchObject({ + status: 'skipped', + reason: 'missing-origin', + }) + }) + + it('classifies an undecodable original as an honest skip', async () => { + const photo = await createPhoto() + await writeFile(join(staticDir(), photo.filename), Buffer.from('not an image')) + + expect(await generateArchivePhotoGrade(payload, { filename: photo.filename })).toMatchObject({ + status: 'skipped', + reason: 'corrupt-origin', + }) + }) + + it('serves the stored AVIF to clients that accept it and the JPEG fallback otherwise', async () => { + const photo = await createPhoto() + await approve(photo.id) + await settleGrade(photo.filename) + + const avif = await getArchivePhotoMedia( + new Request(`http://localhost/fotos/${photo.id}/midia?tamanho=grade`, { + headers: { accept: 'image/avif,image/webp,*/*' }, + }), + { params: Promise.resolve({ id: String(photo.id) }) }, + ) + expect(avif.status).toBe(200) + expect(avif.headers.get('Content-Type')).toBe('image/avif') + expect(avif.headers.get('Vary')).toBe('Accept') + expect(avif.headers.get('Cache-Control')).toContain('no-store') + expect(avif.headers.get('X-Robots-Tag')).toBe('noindex') + expect(avif.headers.get('Content-Disposition')).toContain('inline') + // A stored hit never schedules the heal. + expect(afterMock).not.toHaveBeenCalled() + + const jpeg = await getArchivePhotoMedia( + new Request(`http://localhost/fotos/${photo.id}/midia?tamanho=grade`), + { params: Promise.resolve({ id: String(photo.id) }) }, + ) + expect(jpeg.status).toBe(200) + expect(jpeg.headers.get('Content-Type')).toBe('image/jpeg') + expect(jpeg.headers.get('Vary')).toBe('Accept') + }) + + it('falls back to the on-the-fly JPEG when the grade is missing', async () => { + const photo = await createPhoto() + await approve(photo.id) + await settleGrade(photo.filename) + await rm(gradePathOf(photo.filename), { force: true }) + + const response = await getArchivePhotoMedia( + new Request(`http://localhost/fotos/${photo.id}/midia?tamanho=grade`, { + headers: { accept: 'image/avif' }, + }), + { params: Promise.resolve({ id: String(photo.id) }) }, + ) + expect(response.status).toBe(200) + expect(response.headers.get('Content-Type')).toBe('image/jpeg') + expect(response.headers.get('Vary')).toBe('Accept') + // The miss schedules the lazy heal for the next request. + expect(afterMock).toHaveBeenCalledTimes(1) + }) +}) diff --git a/tests/unit/archiveIntegrityPlan.unit.spec.ts b/tests/unit/archiveIntegrityPlan.unit.spec.ts index d8af9b84..bfb9654c 100644 --- a/tests/unit/archiveIntegrityPlan.unit.spec.ts +++ b/tests/unit/archiveIntegrityPlan.unit.spec.ts @@ -8,12 +8,12 @@ import { archiveIntegrityReportStamp, formatArchiveIntegrityRepairReport, formatArchiveIntegrityScanReport, - mapWithConcurrency, parseArchiveIntegrityCliArgs, parseArchiveIntegrityOnly, summarizeArchiveIntegrityRepair, summarizeArchiveIntegrityScan, } from '../../scripts/lib/archiveIntegrityPlan.mjs' +import { mapWithConcurrency } from '../../scripts/lib/cli.mjs' // C246 — the pure parser/summary of `pnpm archive:integrity`: the read-only // default, the bounded-concurrency map and the honest accounting of the sweep diff --git a/tests/unit/archivePhotoThumbnail.unit.spec.ts b/tests/unit/archivePhotoThumbnail.unit.spec.ts new file mode 100644 index 00000000..c0139dc4 --- /dev/null +++ b/tests/unit/archivePhotoThumbnail.unit.spec.ts @@ -0,0 +1,41 @@ +// @vitest-environment node + +import { describe, expect, it } from 'vitest' + +import { + ARCHIVE_PHOTO_GRADE_EXTENSION, + ARCHIVE_PHOTO_GRADE_MIME_TYPE, + ARCHIVE_PHOTO_GRADE_QUALITY, + archivePhotoGradeFilename, +} from '@/lib/archivePhotoThumbnail' + +// C248 — the pure naming rules of the stored grade: the deterministic sibling +// key the route, the generation utility and the backfill CLI share. The +// extension swap is the whole contract; a collision with the original key or a +// nondeterministic name would silently break the discovery by exact key. + +describe('archive photo grade naming (C248)', () => { + it('derives the sibling key by replacing the original extension', () => { + expect(archivePhotoGradeFilename('flickr-123.jpg')).toBe('flickr-123-grade.avif') + expect(archivePhotoGradeFilename('flickr-123.jpeg')).toBe('flickr-123-grade.avif') + expect(archivePhotoGradeFilename('flickr-123.png')).toBe('flickr-123-grade.avif') + expect(archivePhotoGradeFilename('flickr-abc_def-9.JPG')).toBe('flickr-abc_def-9-grade.avif') + }) + + it('keeps the whole name as the base when there is no usable extension', () => { + expect(archivePhotoGradeFilename('flickr-123')).toBe('flickr-123-grade.avif') + expect(archivePhotoGradeFilename('.hidden')).toBe('.hidden-grade.avif') + expect(archivePhotoGradeFilename('flickr-1.2.jpg')).toBe('flickr-1.2-grade.avif') + }) + + it('never derives the original key itself', () => { + const original = 'flickr-999.jpg' + expect(archivePhotoGradeFilename(original)).not.toBe(original) + }) + + it('pins the AVIF container and the repo quality ruler', () => { + expect(ARCHIVE_PHOTO_GRADE_MIME_TYPE).toBe('image/avif') + expect(ARCHIVE_PHOTO_GRADE_EXTENSION).toBe('.avif') + expect(ARCHIVE_PHOTO_GRADE_QUALITY).toBe(60) + }) +}) diff --git a/tests/unit/archiveThumbnailPlan.unit.spec.ts b/tests/unit/archiveThumbnailPlan.unit.spec.ts new file mode 100644 index 00000000..0cca1c78 --- /dev/null +++ b/tests/unit/archiveThumbnailPlan.unit.spec.ts @@ -0,0 +1,130 @@ +// @vitest-environment node + +import { describe, expect, it } from 'vitest' + +import { + ARCHIVE_THUMBNAIL_DEFAULT_CONCURRENCY, + ARCHIVE_THUMBNAIL_DEFAULT_OUT_DIR, + archiveThumbnailEligibleRows, + parseArchiveThumbnailCliArgs, + parseArchiveThumbnailOnly, + summarizeArchiveThumbnailProbe, + summarizeArchiveThumbnailRun, +} from '../../scripts/lib/archiveThumbnailPlan.mjs' + +// C248 — the pure parser/accounting of `pnpm archive:thumbnails`: the +// read-only default, the approved-only queue and the honest summary of the +// generation run (already present is a skip, not work; only real failures +// deserve a retry). + +describe('parseArchiveThumbnailCliArgs', () => { + it('defaults to the read-only plan and parses the options', () => { + expect(parseArchiveThumbnailCliArgs([])).toEqual({ + apply: false, + verify: false, + only: null, + limit: null, + concurrency: ARCHIVE_THUMBNAIL_DEFAULT_CONCURRENCY, + out: ARCHIVE_THUMBNAIL_DEFAULT_OUT_DIR, + help: false, + }) + + const apply = parseArchiveThumbnailCliArgs([ + '--apply', + '--only', + '80,83,80', + '--limit', + '5', + '--concurrency', + '2', + '--out', + 'x', + ]) + expect(apply).toEqual({ + apply: true, + verify: false, + only: [80, 83], + limit: 5, + concurrency: 2, + out: 'x', + help: false, + }) + + expect(parseArchiveThumbnailCliArgs(['--verify']).verify).toBe(true) + expect(parseArchiveThumbnailCliArgs(['--help']).help).toBe(true) + expect(parseArchiveThumbnailCliArgs(['-h']).help).toBe(true) + }) + + it('fails closed on bad values and unknown args', () => { + expect(() => parseArchiveThumbnailCliArgs(['--only'])).toThrow(/faltou valor/) + expect(() => parseArchiveThumbnailCliArgs(['--only', ''])).toThrow(/--only/) + expect(() => parseArchiveThumbnailCliArgs(['--only', '80,abc'])).toThrow(/--only/) + expect(() => parseArchiveThumbnailCliArgs(['--only', '0'])).toThrow(/--only/) + expect(() => parseArchiveThumbnailCliArgs(['--limit', '0'])).toThrow(/--limit/) + expect(() => parseArchiveThumbnailCliArgs(['--limit', 'x'])).toThrow(/--limit/) + expect(() => parseArchiveThumbnailCliArgs(['--concurrency', '0'])).toThrow(/--concurrency/) + expect(() => parseArchiveThumbnailCliArgs(['--out', '../etc'])).toThrow(/--out/) + expect(() => parseArchiveThumbnailCliArgs(['--apply=1'])).toThrow(/argumento desconhecido/) + }) + + it('refuses the contradictory modes together', () => { + expect(() => parseArchiveThumbnailCliArgs(['--apply', '--verify'])).toThrow( + /mutuamente exclusivos/, + ) + }) + + it('parses --only without duplicates and refuses an empty selection', () => { + expect(parseArchiveThumbnailOnly('80, 83,80')).toEqual([80, 83]) + expect(() => parseArchiveThumbnailOnly(' ')).toThrow(/--only/) + }) +}) + +describe('archiveThumbnailEligibleRows', () => { + it('keeps only approved rows with a stored filename', () => { + const rows = [ + { id: 1, flickrId: '1', filename: 'a.jpg', publicationStatus: 'approved' }, + { id: 2, flickrId: '2', filename: 'b.jpg', publicationStatus: 'draft' }, + { id: 3, flickrId: '3', filename: 'c.jpg', publicationStatus: 'removed' }, + { id: 4, flickrId: '4', filename: null, publicationStatus: 'approved' }, + { id: 5, flickrId: '5', filename: null, publicationStatus: 'draft' }, + ] + + expect(archiveThumbnailEligibleRows(rows)).toEqual([ + { id: 1, flickrId: '1', filename: 'a.jpg' }, + ]) + expect(archiveThumbnailEligibleRows([])).toEqual([]) + }) +}) + +describe('summarizeArchiveThumbnailProbe', () => { + it('counts present, missing and names the missing ids', () => { + const summary = summarizeArchiveThumbnailProbe([ + { id: 1, flickrId: '1', present: true }, + { id: 2, flickrId: '2', present: false }, + { id: 3, flickrId: '3', present: false }, + ]) + expect(summary).toEqual({ eligible: 3, present: 1, missing: 2, missingIds: [2, 3] }) + }) +}) + +describe('summarizeArchiveThumbnailRun', () => { + it('separates generated, skipped-by-reason and failures', () => { + const summary = summarizeArchiveThumbnailRun([ + { id: 1, outcome: { status: 'generated', bytes: 100 } }, + { id: 2, outcome: { status: 'generated', bytes: 50 } }, + { id: 3, outcome: { status: 'skipped', reason: 'already-present' } }, + { id: 4, outcome: { status: 'skipped', reason: 'missing-origin' } }, + { id: 5, outcome: { status: 'skipped', reason: 'corrupt-origin' } }, + { id: 6, outcome: { status: 'failed', reason: 'upload: timeout' } }, + ]) + + expect(summary).toEqual({ + eligible: 6, + generated: 2, + skipped: 3, + failed: 1, + bytes: 150, + skippedByReason: { 'already-present': 1, 'missing-origin': 1, 'corrupt-origin': 1 }, + }) + }) +}) diff --git a/tests/unit/privateMedia.unit.spec.ts b/tests/unit/privateMedia.unit.spec.ts index 86274bd8..982f67f7 100644 --- a/tests/unit/privateMedia.unit.spec.ts +++ b/tests/unit/privateMedia.unit.spec.ts @@ -3,6 +3,7 @@ import { describe, expect, it } from 'vitest' import { + acceptsAvif, isPrivateMediaMissingError, PRIVATE_MEDIA_FALLBACK_MIME_TYPE, privateMediaContentDisposition, @@ -120,6 +121,28 @@ describe('private media response rules (C193/C199)', () => { }) }) +describe('private media AVIF negotiation (C248)', () => { + it('accepts only an explicit image/avif token with a positive quality', () => { + expect(acceptsAvif('image/avif')).toBe(true) + expect(acceptsAvif('IMAGE/AVIF')).toBe(true) + expect(acceptsAvif('text/html, image/avif')).toBe(true) + expect(acceptsAvif('image/avif;q=0.5')).toBe(true) + expect(acceptsAvif('image/avif; q=0.5, image/webp')).toBe(true) + expect(acceptsAvif('image/avif;q=0')).toBe(false) + expect(acceptsAvif('image/avif;q=0.000')).toBe(false) + }) + + it('keeps the JPEG fallback for headers that never opted in', () => { + expect(acceptsAvif(null)).toBe(false) + expect(acceptsAvif(undefined)).toBe(false) + expect(acceptsAvif('')).toBe(false) + expect(acceptsAvif('*/*')).toBe(false) + expect(acceptsAvif('image/*')).toBe(false) + expect(acceptsAvif('image/jpeg, image/webp')).toBe(false) + expect(acceptsAvif('image/avifjunk')).toBe(false) + }) +}) + describe('private media missing-object classification (C246)', () => { it('recognizes the S3 and local-disk absence shapes', () => { expect(isPrivateMediaMissingError({ name: 'NoSuchKey' })).toBe(true)