feat(C232): pré-catalogar o acervo de fotos com IA local e curadoria que vence

Collection archivePhoto ganha o grupo catalog (legenda/descrição/cena/texto
visível/temas/pessoas/município/origem/catalogedAt), curatedFields (o que a
assessoria edita nunca é sobrescrito), takenOn e searchText com índice GIN trgm
(migration aditiva add_archive_photo_catalog); hook beforeValidate deriva o
índice e marca curadoria só em edição humana; pipeline transacional baixa os
bytes privados, prepara JPEG <=1024px com sharp e mescla o VLM local com o
gazetteer/catálogo curado (pessoas só do TEXTO, nunca de rosto); CLI
pnpm archive:catalog (plan/--apply/--verify, recibo honesto, idempotente por
catalogedAt) com engine OpenAI-compatible na rede privada atrás de guarda de
host (ARCHIVE_VISION_ALLOW_REMOTE é o escape); client HTTP com retry/timeout e
parser zod fail-closed; resolver de município extraído para owner compartilhado
com o catálogo de peças delegando; unit/int novos e runbook staging-primeiro.
This commit is contained in:
Francisco Solla 2026-09-26 10:49:37 -03:00
parent 3ccb1aea9b
commit 0a8dabc192
28 changed files with 23743 additions and 32 deletions

View file

@ -105,3 +105,14 @@ PORTAL_TRANSPARENCIA_API_KEY=
# read by the CLI at run time and never stored in receipts.
FLICKR_API_KEY=
FLICKR_USER_ID=
# C232 — vision engine of the photo-archive cataloguing (`pnpm archive:catalog`).
# Any OpenAI-compatible server (Ollama, llama.cpp, vLLM) on the PRIVATE network:
# the citizens' photos must not leave it (a public host is refused unless
# ARCHIVE_VISION_ALLOW_REMOTE=1 is declared). Example: an Ollama box on the LAN
# or tailnet — ARCHIVE_VISION_BASE_URL=http://100.94.122.26:11434/v1.
# ARCHIVE_VISION_API_KEY is only for servers that require a key; never stored in
# receipts, and the endpoint is only called by `--apply`.
ARCHIVE_VISION_BASE_URL=
ARCHIVE_VISION_MODEL=
ARCHIVE_VISION_API_KEY=

2
.gitignore vendored
View file

@ -86,6 +86,8 @@ next-env.d.ts
/data/flickr/
# C229 — speech semantic index reports (pnpm acervo:index)
/data/acervo-index/
# C232 — photo-archive cataloguing receipts (pnpm archive:catalog)
/data/archive/
# Solla × Ceuci report HTML cache (scripts/build-solla-ceuci-salvador-report.mjs)
/data/solla-ceuci-salvador/

View file

@ -0,0 +1 @@
**C232 — catalogação com IA do acervo de fotos (2026-09-26):** as 6,5k fotos do acervo privado (`archivePhoto`, C231) ganham ficha pré-catalogada revisável no admin: legenda/descrição pt-BR, atividade/cena (vocabulário fechado de 11 valores), texto visível (OCR do próprio VLM), temas do vocabulário existente (`SPEECH_TOPICS`), município pelo gazetteer (regra conservadora: Salvador/zona ambíguo → null, resolver extraído para `src/utilities/municipality/municipalityMentionResolver.ts` com o catálogo de peças delegando) e pessoas públicas **só do catálogo curado, lidas do texto — nunca inferidas de rosto**. O engine de visão é um endpoint OpenAI-compatible local (`ARCHIVE_VISION_BASE_URL`/`MODEL`/`API_KEY` — Ollama/llama.cpp/vLLM; nenhum fornecedor pinado) atrás de guarda falha-fechado de host privado (`isPrivateVisionHost`/`assertLocalVisionEndpoint` em `scripts/lib/cli.mjs`, escape explícito `ARCHIVE_VISION_ALLOW_REMOTE=1` com o escopo registrado no recibo): PII/rostos não saem da rede sem decisão; terceiro/híbrido rejeitados com gatilho de revisita. A curadoria vence a IA pelo mesmo trio do C230 (`ARCHIVE_PHOTO_CURATED_FIELDS` no lib + hook `beforeValidate` que marca campo alterado só quando há `req.user` e não é escrita do sistema + fresh-read dentro da transação em `src/utilities/flickr/archivePhotoCatalog.ts`, que baixa os bytes privados via `downloadPrivateMediaToFile`, prepara JPEG ≤1024px com sharp, injeta o analisador e grava só campo não curado); `catalog.source` = `ai | metadata | none` (nada a propor é estado final honesto) e `catalog.catalogedAt` é a chave de idempotência. O passe é o CLI de manutenção `pnpm archive:catalog` (plan default / `--apply` com `ARCHIVE_CATALOG_CONFIRM=1` + `TEQO_ENV` + 4 `S3_*` / `--verify` read-only que sai 1 com pendentes; `--limit` canário; recibo JSON datado em `data/archive/reports/`), com client HTTP com retry/timeout em `scripts/lib/archiveVisionApi.mjs` e parser zod fail-closed no lib. A busca interna é o admin stock: `searchText` normalizado com índice GIN trgm + `takenOn` (derivado do wall clock do Flickr, meio-dia UTC) + facetas filtráveis (município/cena/temas/pessoas/data/origem); `alt` é refinado a partir da caption só quando nunca curado. Migration aditiva `add_archive_photo_catalog` (grupo `catalog`, `curatedFields`, `takenOn`, `searchText` + índices). Testes: unit do contrato puro/parser/merge, do client (fetch mockado), do parser/recibo e dos guardas por subprocesso; int sobre `teqo_test` (catálogo grava e deriva o índice, curadoria vence, falha não escreve e retenta, skip por `catalogedAt`, fontes ai/metadata/none, marcação só em edição humana); sem e2e novo (superfície admin stock, OPS72). Runbook staging-primeiro com canário em `docs/ops/teqo-1313-deploy.md`. Impl plan: `docs/plans/acervo-fotos-catalogacao-ia-impl.md`.

View file

@ -1375,6 +1375,60 @@ apagar as linhas do `archivePhoto` no admin (o objeto correspondente no bucket
pode ser removido à parte); reexecutar a ingestão recria as mesmas chaves de
objeto.
## C232 — catalogação com IA do acervo de fotos
O comando `pnpm archive:catalog` pré-cataloga as fotos do acervo (`archivePhoto`)
com um engine de visão LOCAL: legenda/descrição, atividade/cena, texto visível,
temas do vocabulário existente, município (gazetteer) e pessoas públicas (só do
catálogo curado, lidas do TEXTO — nunca de rosto). A curadoria da assessoria
vence: campo editado no admin entra em "Campos curados" e a catalogação nunca
sobrescreve; reexecutar converge ("processadas / puladas / falharam com motivo")
e só as pendentes são processadas. Nada é publicado.
O engine é qualquer servidor OpenAI-compatible na **rede privada** (Ollama,
llama.cpp, vLLM). Exemplo: servir o modelo na workstation com GPU (o mesmo padrão
do ML do Immich no tailnet) e apontar o container para lá:
```bash
# na workstation (GPU), ex. com Ollama:
ollama serve # porta 11434, alcançável pelo tailnet
ollama pull qwen2.5vl:7b
# no homeserver:
ssh homeserver
cd ~/stack
# 0) plano (dry-run; lê a fila, não chama o engine nem escreve):
docker compose --profile maintenance run --rm \
-e ARCHIVE_VISION_BASE_URL=http://100.94.122.26:11434/v1 \
-e ARCHIVE_VISION_MODEL=qwen2.5vl:7b \
teqo-staging-migrate pnpm archive:catalog --limit 5
# 1) canário no staging (exige TEQO_ENV + confirmação; as 4 S3_* são obrigatórias):
docker compose --profile maintenance run --rm \
-e TEQO_ENV=staging -e ARCHIVE_CATALOG_CONFIRM=1 \
-e ARCHIVE_VISION_BASE_URL=http://100.94.122.26:11434/v1 \
-e ARCHIVE_VISION_MODEL=qwen2.5vl:7b \
teqo-staging-migrate pnpm archive:catalog --apply --limit 5
# 2) validação no staging e só então produção (TEQO_ENV=production, serviço teqo-1313-migrate)
# 3) inventário a qualquer momento (read-only, sem engine):
docker compose --profile maintenance run --rm \
teqo-staging-migrate pnpm archive:catalog --verify
```
Os recibos JSON ficam em `data/archive/reports/` do container — monte um volume
(`-v /srv/archive-reports:/app/data/archive`) para preservá-los. Guardas: fora de
teste o `TEQO_ENV=staging|production` é obrigatório com o **nome exato** do banco
(`teqo_staging`/`teqo_1313`); `ALLOW_REMOTE_DB` é recusado; sem as 4 `S3_*` o
`--apply` recusa (o acervo não estaria acessível no alvo); `ARCHIVE_CATALOG_CONFIRM=1`
é exigido em alvo não-local; e o endpoint do engine precisa estar na rede privada
— host público é recusado e exige `ARCHIVE_VISION_ALLOW_REMOTE=1` explícito
(registrado no recibo com host/modelo e o escopo). Falha por foto é isolada e
nomeada (nada é escrito; a próxima execução retenta); o lote de 6,5k é sequencial
(~4–10 h). A chave do engine (quando houver) nunca entra em recibo/log.
Rollback: a migration `add_archive_photo_catalog` é aditiva; o rollback funcional
é limpar o grupo `catalog`/`curatedFields` no admin. Reexecutar nunca duplica nem
sobrescreve curadoria.
## Referências
- `scripts/deploy-homeserver.sh` — o script (fonte da verdade do fluxo; parametrizado por `TEQO_ENV`)

View file

@ -0,0 +1,188 @@
# Impl: C232 — Catalogação com IA do acervo de fotos
Status: aprovado
Atualizado em: 2026-09-26
Issue: #1367
Intenção: docs/plans/acervo-fotos-catalogacao-ia.md
Appetite restante: herdado — ~2 dias eng + tempo de processamento do lote (6,5k fotos)
## Leitura da intenção
- **Outcome:** cada foto do acervo C231 ganha ficha pré-catalogada revisável (legenda/descrição pt-BR, atividade/cena, local, pessoas do catálogo curado, texto visível, temas, descrição semântica); a busca interna acha por local/data/atividade/pessoa/termo; a curadoria humana vence a IA; o passe é em lote não interativo com recibo honesto e reexecução idempotente; nada é público e nenhum nome é inferido de rosto.
- **O que NÃO negociar:** PII/rostos de cidadãos não vão a terceiro sem decisão explícita e contrato; provider reservado aos agentes de design (`openai/*`, OPS123) não entra no app; pessoas só do catálogo curado e com confirmação humana; rosto é presença/tag de cena (biometria é C234); sem segundo engine de catalogação, sem segundo cadastro de pessoa, sem edição de imagem; leitura do acervo pelo mesmo gate da Comunicação (`src/utilities/access/archivePhotos.ts:18-23`).
- **O que reavaliar:** a hipótese da intenção aponta `src/utilities/content/` e `src/utilities/ai/` como áreas prováveis; a engenharia reavalia e escolhe `src/utilities/flickr/` (dono do archivePhoto server-side, C231) + `src/lib` puro + `scripts/lib` para o client HTTP, **sem** reusar o job de peças (input é imagem, não transcrição). A hipótese "embedding para busca futura" fica adiada com gatilho (só a descrição textual agora).
## Abordagem recomendada
```mermaid
flowchart LR
A[archivePhoto C231<br/>catalog.catalogedAt nulo] --> B[CLI archive:catalog --apply]
B --> C[baixa bytes privados<br/>downloadPrivateMediaToFile]
C --> D[sharp ≤1024px JPEG<br/>só para o engine]
D --> E[analisador injetável<br/>VLM local OpenAI-compatible]
E --> F[validação pura pt-BR:<br/>cena · temas · pessoas do catálogo]
F --> G[determinístico de texto:<br/>município · temas · pessoas]
G --> H[transação: fresh-read curatedFields<br/>escreve só não curado + source/catalogedAt]
H --> I[(admin: busca searchText/trgm<br/>filtros local/data/atividade/pessoa)]
B -.-> R[recibo JSON: processadas/puladas/falharam com motivo]
```
**Opções consideradas:** A) pipeline local/self-hosted em CLI de manutenção (recomendada) | B) API de visão de terceiro | C) job in-app com fila.
**Recomendação:** A — o lote não é interativo e roda em casa; o engine é um endpoint OpenAI-compatible configurado por env (Ollama/llama.cpp/vLLM na workstation com GPU, padrão de serviço via tailnet já usado pelo ML do Immich), então nenhuma foto sai da rede privada e nenhum fornecedor fica pinado no código.
**Rejeitadas:** B porque manda foto de cidadão para terceiro sem decisão explícita e contrato (guardrail de produto); C porque ainda exterioriza a imagem e cria dois caminhos para o mesmo risco.
### Componentes / mudanças
**Novos — contrato puro (sem I/O, sem `server-only`; mesmo molde de `src/lib/archivePhoto.ts:1-11`)**
- **`ARCHIVE_PHOTO_SCENES` / `isArchivePhotoScene` / `archivePhotoSceneLabel`** (`src/lib/archivePhotoCatalog.ts`): vocabulário fechado de atividade/cena com labels pt-BR — `plenaria` (Plenária), `audiencia` (Audiência), `reuniao` (Reunião), `evento` (Evento), `mobilizacao` (Mobilização), `visita` (Visita), `entrevista` (Entrevista), `discurso` (Discurso), `retrato` (Retrato), `gabinete` (Gabinete), `outro` (Outro).
- **`ARCHIVE_PHOTO_CURATED_FIELDS` + labels** (`src/lib/archivePhotoCatalog.ts`): `alt`, `caption`, `description`, `scene`, `visibleText`, `hasPeople`, `themes`, `people`, `municipality` — espelha `CONTENT_PIECE_CURATED_FIELDS` (`src/lib/contentPiece.ts:111-137`), um literal por nome de campo persistido.
- **`ARCHIVE_PHOTO_CATALOG_SOURCES`** (`ai | metadata | none`) + labels pt-BR.
- **`archivePhotoThemesFrom`** (`src/lib/archivePhotoCatalog.ts`): resolve tokens contra `resolveSpeechTopic` fail-closed (`src/lib/speechFacets.ts:83-88`), deduplica e capa; nunca inventa slug.
- **`archivePhotoPeopleFromCatalog`** (`src/lib/archivePhotoCatalog.ts`): só nomes que resolvem em `resolvePublicFigureName` (`src/lib/publicFigureCatalog.ts:77-85`); valor não resolvido é descartado, nunca vira nome.
- **`matchPublicFigureMentions`** (`src/lib/archivePhotoCatalog.ts`): menção exata de nome do catálogo no texto (molde `resolveContentPieceInstitution`, `src/utilities/content/contentPieceCataloging.ts:189-201`) para o caminho determinístico `metadata`.
- **`archivePhotoSearchText`** (`src/lib/archivePhotoCatalog.ts`): haystack normalizado com `normalizeForSearch` (`src/lib/speechSearch.ts:11-17`; molde `contentPieceSearchText`, `src/lib/contentPiece.ts:337-352`) — alt, título, legenda, descrição (Flickr + catálogo), cena, texto visível, temas, pessoas, município, álbuns e tags.
- **`archivePhotoTakenOn`** (`src/lib/archivePhotoCatalog.ts`): `YYYY-MM-DD` do wall clock do Flickr (`archivePhotoTakenAt`, `src/lib/archivePhoto.ts:123-127`), fixado ao meio-dia UTC para o admin não exibir o dia anterior.
- **`changedArchivePhotoCuratedFields`** (`src/lib/archivePhotoCatalog.ts`): diff do hook do admin (arrays normalizados; esvaziar conta como decisão).
- **`buildArchivePhotoCatalogWrite`** (`src/lib/archivePhotoCatalog.ts`): merge determinístico que só preenche campo fora de `curatedFields`; `alt` refina a partir da caption só quando não curado (C231 delegou "C232 refina", `src/lib/archivePhoto.ts:118-120`); devolve `{ data, source }`.
- **`parseArchiveVisionOutput` / `buildArchiveVisionPrompt` / `ARCHIVE_VISION_SYSTEM_PROMPT`** (`src/lib/archivePhotoCatalog.ts`): parser zod fail-closed → `ArchivePhotoSuggestion | null`; prompt pt-BR com o vocabulário fechado (cenas, `SPEECH_TOPICS` `src/lib/speechFacets.ts:14-33`, nomes do `publicFigureCatalog` `src/lib/publicFigureCatalog.ts:40-51`) e a regra "nunca inferir nome de rosto; só nomes do catálogo; responder só JSON".
**Novos — server-side Payload**
- **`resolveMentionedMunicipalityId`** (`src/utilities/municipality/municipalityMentionResolver.ts`): extração do lookup de `resolveContentPieceMunicipalityId` (`src/utilities/content/contentPieceCataloging.ts:159-182`) para owner compartilhado na subpasta de domínio já existente; mantém a regra conservadora (só cidade com exatamente 1 entrada; Salvador ambíguo → `null`).
- **`ArchivePhotoAnalyzer` / `listArchivePhotoCatalogQueue` / `catalogArchivePhoto`** (`src/utilities/flickr/archivePhotoCatalog.ts`, `import 'server-only'`; o laço por foto e o recibo ficam no CLI): pipeline por foto com analisador injetável; baixa os bytes privados com `downloadPrivateMediaToFile` + `resolvePrivateMediaStaticDir` (`src/utilities/privateMedia/privateMediaResponse.ts:219-249,43-51`); prepara o JPEG ≤1024px com `sharp` (`package.json:150`); roda o gazetteer determinístico (`matchMunicipalityMentions` `src/lib/speechGazetteer.ts:396-431`, `classifySpeechByGazetteer` `:449-476`); dentro de `withPayloadTransaction` (`src/utilities/payloadTransaction.ts:86-113`) relê `curatedFields`/`catalog.catalogedAt` e escreve só campo não curado + `catalog.source`/`catalog.catalogedAt` (molde `contentPieceJob.ts:285-323`); `context: { archivePhotoCatalog: true }` e `overrideAccess: true` com comentário de bypass.
**Novos — scripts/CLI**
- **`createArchiveVisionClient` / `analyzeImage` / `ArchiveVisionError`** (`scripts/lib/archiveVisionApi.mjs`): client HTTP OpenAI-compatible (`POST {base}/chat/completions`, imagem em `data:` URL base64), `fetchImpl` injetável, timeout ~90s, retry/backoff só em transitório (429/5xx/rede) e erros nomeados — molde `scripts/lib/flickrApi.mjs:27-46,60-124`; nunca loga a chave; entra em `SCRIPTS_SPEC_PINNED` (`scripts/lib/test-affected-core.mjs:31-110`).
- **`parseArchiveCatalogCliArgs` / summaries / `formatArchiveCatalogReport`** (`scripts/lib/archiveCatalogPlan.mjs`): parser dos modos e recibo (molde `scripts/lib/flickrPlan.mjs:48-90,337-373`); também pinado.
- **`scripts/catalog-archive-photos.mjs`** (+ script `pnpm archive:catalog`, molde `package.json:37`): plan default / `--apply` / `--verify`, `--limit` canário, `--out`, `--help`; guardas reusados de `scripts/lib/cli.mjs` (`assertWriteConfirm` com `ARCHIVE_CATALOG_CONFIRM`, `assertEnvironmentDatabaseTarget`, `mirroredMediaRequired` `:312-320,238-301,197-202`) + `assertLocalVisionEndpoint` novo; loop com falha isolada e estágio nomeado (molde `scripts/ingest-flickr-archive.mjs:154-231`); recibo JSON datado em `data/archive/reports/`; `--verify` sai 1 com fotos pendentes.
**Novos — migration/testes/ops**
- **Migration `add_archive_photo_catalog`** (`src/migrations/20260926_HHMMSS_add_archive_photo_catalog.ts` + `.json` + registro em `src/migrations/index.ts`, hoje último em `:89`): grupo `catalog`, `curated_fields`, `taken_on`, `search_text`, índices e GIN trgm (molde `20260923_032714_add_content_piece.ts:5,100`).
- **Testes:** `tests/unit/archivePhotoCatalog.unit.spec.ts`, `tests/unit/archiveVisionApi.unit.spec.ts`, `tests/unit/archiveCatalogPlan.unit.spec.ts`, `tests/unit/visionEndpoint.unit.spec.ts`, `tests/unit/archiveCatalogCli.unit.spec.ts` (guardas por subprocesso); `tests/int/archivePhotoCatalog.int.spec.ts` sobre a fixture JPEG `tests/helpers/archivePhotoFixture.ts`.
- **Ops:** seção C232 em `docs/ops/teqo-1313-deploy.md` (após `:1299-1339`), `ARCHIVE_VISION_*` em `.env.example` (`:102-107` é o bloco C231), `/data/archive/` em `.gitignore` (`:86` é o `/data/flickr/`), `docs/changelog/2026-09-26-c232.md`.
**Edits**
- **`markArchivePhotoCuratedFields` / `deriveArchivePhotoCatalogIndex`** (`src/collections/ArchivePhoto.ts`, hooks inline como manda a convenção): marca campo alterado quando `req.user` existe e `req.context.archivePhotoCatalog` está ausente (precedente de contexto: `src/utilities/googleCalendarSync.ts:508`; leitura do contexto de hook: `src/collections/Activity.ts:79`); deriva `takenOn` e `searchText` em `beforeValidate` mesclando `data` com `originalDoc` como `ContentPiece.ts:164-240`. Access intocado (`:32-37`).
- **Campos novos em `ArchivePhoto.ts`:** grupo `catalog` (`caption` text, `description` textarea, `scene` select index, `visibleText` textarea, `hasPeople` checkbox, `themes` select hasMany index com `SPEECH_TOPICS`, `people` text hasMany, `municipality` relationship, `source` select, `catalogedAt` date index); top-level `curatedFields` (select hasMany readOnly, mesmo trio C230) e `takenOn` (date readOnly index); `searchText` (textarea readOnly).
- **Admin do `ArchivePhoto.ts`:** `useAsTitle: 'alt'`, `defaultColumns: ['alt', 'takenOn', 'catalog.scene', 'catalog.municipality', 'catalog.source']`, `listSearchableFields: ['searchText']` (opção suportada no Payload 3.82, `node_modules/payload/dist/collections/config/types.d.ts:405`, implementada com `like`).
- **`resolveContentPieceMunicipalityId`** (`src/utilities/content/contentPieceCataloging.ts:159-182`): passa a delegar ao resolver compartilhado, mesma assinatura e comportamento (spec atual `tests/unit/contentPieceCataloging.unit.spec.ts:56-83` continua verde).
- **`assertLocalVisionEndpoint`** (`scripts/lib/cli.mjs`): host privado (loopback/RFC1918/CGNAT `100.64/10`/`.local`/`.internal`/host sem ponto) com escape explícito `ARCHIVE_VISION_ALLOW_REMOTE=1` via `isTruthyEnv`; erro nomeado cita o escape.
### Dados → forma (se aplicável)
Não aplicável — a intenção declara que catalogação não é métrica. Não há score/confiança numérico na ficha; o recibo do lote é operacional (processadas/puladas/falharam por motivo e contagem por `source`), nunca um agregado de leitura.
## Decisões de engenharia
### D1 — Engine de visão
- **Opções:** A) VLM local/self-hosted atrás de um endpoint OpenAI-compatible configurado por env | B) API de terceiro (DeepInfra, já contratada para ASR/chat) | C) híbrido local+terceiro | D) modelo ONNX in-process (transformers.js).
- **Recomendação:** A — `ARCHIVE_VISION_BASE_URL` + `ARCHIVE_VISION_MODEL` obrigatórios no `--apply` e `ARCHIVE_VISION_API_KEY` opcional (servidores locais normalmente não têm chave); exemplo operacional: Ollama/llama.cpp/vLLM na workstation com GPU, alcançável pelo tailnet no mesmo padrão do ML do Immich (`IMMICH_MACHINE_LEARNING_URL=http://100.94.122.26:3003`). Sem vendor pin: o contrato é o protocolo, não o fornecedor.
- **Rejeitadas:** B porque manda PII/rostos de cidadãos a terceiro sem decisão explícita e contrato (guardrail de produto); C porque ainda exterioriza a foto sensível e paga a complexidade de dois caminhos sem ganho de v1; D porque a qualidade para 6,5k fotos e a dependência pesada não se pagam, e o repo não tem esse runtime.
- **Gatilho de revisita:** se a qualidade local não fechar o aceite no canário, reabre B/C com decisão explícita de PII — é o que a intenção manda.
### D2 — Guardrail de localidade do engine
- **Opções:** A) `assertLocalVisionEndpoint` em `scripts/lib/cli.mjs` + unit da função pura | B) só documentar a env no runbook | C) checar apenas dentro do parser de argv.
- **Recomendação:** A — host privado (loopback, RFC1918, CGNAT `100.64/10` — o tailnet da workstation, sufixos `.local`/`.internal`, host sem ponto como `host.docker.internal`) com escape explícito `ARCHIVE_VISION_ALLOW_REMOTE=1`; o `--apply` soma engine configurado + `ARCHIVE_CATALOG_CONFIRM=1` fora de dev local + `TEQO_ENV`/S3 (guardas C231, `scripts/ingest-flickr-archive.mjs:245-259`). `cli.mjs` é o dono dos guardas de CLI (C155/C231).
- **Rejeitadas:** B porque um typo na base URL mandaria foto de cidadão para um endpoint público em silêncio; C porque a guarda não ficaria unit-pinada nem reutilizável pelo runbook.
- **Recibo:** o recibo registra `engineScope: local|remote` (host e modelo, nunca a chave) para o escape nunca ser invisível.
### D3 — Fronteira de módulos
- **Opções:** A) contrato puro em `src/lib/archivePhotoCatalog.ts` + pipeline Payload em `src/utilities/flickr/archivePhotoCatalog.ts` + client HTTP em `scripts/lib/archiveVisionApi.mjs` + CLI `scripts/catalog-archive-photos.mjs` | B) tudo no pipeline `src/utilities/flickr/` (client embutido) | C) job in-app disparado por `after()`/fila.
- **Recomendação:** A — a subpasta `src/utilities/flickr/` já é o dono do archivePhoto server-side (C231, `archivePhotoIngest.ts`), e o client de rede com retry/backoff mora em `scripts/lib/` como o `flickrApi.mjs`; isso mantém o contrato puro testável sem Payload e o pipeline testável com analisador stub. O analisador é **injetado** pela CLI (nada em `src/` importa `scripts/`).
- **Rejeitadas:** B porque esconde o transporte na camada errada e perde o pin de CI dos módulos de script; C porque não existe fila Payload no repo e 6,5k fotos em lote pertencem ao container de manutenção (precedente C231), não ao `after()` de um upload.
- **Depth check:** não há módulo profundo de VLM a reusar; o pipeline reusa `downloadPrivateMediaToFile`, `withPayloadTransaction`, o gazetteer/catálogos existentes e o esqueleto de CLI — nenhum pass-through novo é criado.
### D4 — Schema e curadoria que vence a IA
- **Opções:** A) grupo `catalog` dentro do próprio `archivePhoto` + `curatedFields` espelhando C230 (vocabulário no lib + marcação no hook do admin + fresh-read no writer) | B) reusar o pipeline/job do `contentPiece` com campos novos na peça | C) tratar curadoria por trigger de banco / tabela paralela.
- **Recomendação:** A — a ficha e os campos pertencem à foto (C231 é o dono do shape); o vocabulário fica num literal único no lib (como `CONTENT_PIECE_CURATED_FIELDS`, `src/lib/contentPiece.ts:111-137`), o hook `beforeChange` marca o campo alterado quando `req.user` existe e `!req.context.archivePhotoCatalog`, e o writer re-lê a linha **dentro** da transação e só escreve campo ausente de `curatedFields` (molde `contentPieceJob.ts:285-323`). `alt` entra no vocabulário: o catálogo só o refina quando o humano nunca editou.
- **Rejeitadas:** B porque peça e foto têm inputs e ciclos de vida distintos — seria um segundo engine de catalogação disfarçado; C porque tira a regra do admin e do padrão C230, onde o humano vê o que está curado.
- **Escape de sistema:** a escrita do pipeline usa `context: { archivePhotoCatalog: true }` (precedente `src/utilities/googleCalendarSync.ts:508`) e bypass documentado; a ingestão C231 (sem `req.user`) não marca nada.
### D5 — Semântica de estado e idempotência
- **Opções:** A) `catalog.catalogedAt` (date) como chave de idempotência + `catalog.source = ai | metadata | none` | B) persistir estado `failed` no registro | C) marcar `catalogedAt` só quando `source != none`.
- **Recomendação:** A — `--apply` seleciona só `catalog.catalogedAt` nulo; falha de HTTP/parse **não escreve**, vira falha nomeada no recibo e é retentada na próxima execução; `source=ai` quando o modelo contribuiu com algum campo, `metadata` quando só o determinístico de texto (município/temas/pessoas por menção exata no catálogo) produziu algo, `none` quando nada havia a propor. `none` grava `catalogedAt` porque "nada a propor" é estado final honesto (reprocesso futuro é `--refresh`, adiado). Reexecução reporta "puladas (já catalogadas)" e `--verify` é read-only: total, catalogadas por `source`, pendentes — sai 1 se houver pendente.
- **Rejeitadas:** B porque falha não é estado de catálogo; poluiria a ficha e esconderia que a próxima execução resolve; C porque faria o engine re-chamar a mesma foto para sempre.
- **Concorrência:** o fresh-read na transação também confere `catalogedAt` antes de escrever; nada é duplicado.
### D6 — Busca interna (admin stock) e resolver de município
- **Opções:** A) superfície stock do admin — `useAsTitle`, `defaultColumns`, `listSearchableFields: ['searchText']` e filtros nativos sobre `catalog.municipality`, `catalog.scene`, `catalog.themes`, `catalog.people`, `takenOn`, `catalog.source` + índice GIN trgm | B) list view/fila de revisão própria | C) busca externa.
- **Recomendação:** A — a intenção fixa a superfície existente (Impeccable B, sem tela nova); o índice trgm sobre `searchText` é o mesmo contrato do `ContentPiece` (`20260923_032714_add_content_piece.ts:100`); a leitura continua no gate da Comunicação (`access/archivePhotos.ts` intocado) e nada é exposto ao público.
- **Rejeitadas:** B porque é escopo que só nasce se o gate pedir (o design nasceria no gate); C porque não há caso de uso externo.
- **Sub-decisão registrada — resolver de município:** A) extrair o lookup para `src/utilities/municipality/municipalityMentionResolver.ts` com `contentPieceCataloging.resolveContentPieceMunicipalityId` delegando e o pipeline novo importando o genérico (recomendada) | B) importar `resolveContentPieceMunicipalityId` do módulo de peças no pipeline de fotos | C) duplicar a regra conservadora. **Rejeitadas:** B porque acopla a foto ao módulo de conteúdo e puxa as dependências de IA da peça para um caminho que não as quer; C porque a mesma regra em dois lugares diverge (twin). A preserva a assinatura pública e os testes atuais.
### D7 — Testes e invariantes de CI
- **Opções:** A) unit para todo o puro + client mockado + guardas, e int sobre `teqo_wt232_test` com analisador stub; sem e2e novo (OPS72, superfície admin stock) | B) e2e novo para a curadoria no admin | C) confiar em teste manual do lote.
- **Recomendação:** A — unit de `src/lib/archivePhotoCatalog.ts`, do client (`fetchImpl` mockado: retry em transitório, 4xx terminal, timeout/abort, erro nomeado sem chave), do parser/recibo (`archiveCatalogPlan.mjs`) e da guarda de endpoint; int com fixture JPEG (`tests/helpers/archivePhotoFixture.ts`) e stub: catalogação grava campos, curadoria vence, falha não escreve, `source` ai/metadata/none, `searchText`/`takenOn` derivados, access inalterado (molde `tests/int/archivePhotoIngest.int.spec.ts:69-227`). Os dois novos módulos de `scripts/lib` entram em `SCRIPTS_SPEC_PINNED` (`scripts/lib/test-affected-core.mjs:31-110`) para o invariante `ciSkipInvariants` fechar a closure. Rede de provider real nunca entra na suíte.
- **Rejeitadas:** B porque a superfície é o admin stock e o repo não mantém e2e de admin como cobertura de domínio (OPS72); C porque os guardas e a idempotência são exatamente o que não pode falhar em silêncio no lote.
### D8 — Migração
- **Opções:** A) uma migration aditiva gerada por `pnpm migrate:create add_archive_photo_catalog` (colunas do grupo, join tables de `themes`/`people`/`curatedFields`, índices e GIN trgm) | B) confiar em `push` | C) escrever o SQL à mão.
- **Recomendação:** A — `push: false` em todo lugar (`src/payload.config.ts:173`); com o snapshot reparado pelo C231, o diff gerado deve ser limpo; mesmo assim o diff é **revisado à mão** contra o snapshot antes do commit (nenhum objeto existente recriado), aplicado localmente e nunca reescrito depois de mergeado. `down` derruba colunas/tabelas/índices. O `pnpm build` roda `payload migrate` e o deploy aplica no alvo.
- **Rejeitadas:** B porque é a regra de segurança do repo; C porque o gerador é a fonte do diff e o snapshot agora está confiável.
## Fases verificáveis
1. **Fase 1 — Tracer: contrato puro + schema + curadoria** (quota ~0,5 dia): `src/lib/archivePhotoCatalog.ts`, migration, campos/hooks/admin do `ArchivePhoto.ts`; unit do puro; int de 1 foto com stub (grava, curadoria vence, `source`/`catalogedAt`). Gate: `pnpm gate:fast`.
2. **Fase 2 — Pipeline Payload** (quota ~0,5 dia): `src/utilities/flickr/archivePhotoCatalog.ts` + resolver extraído com delegação; int completo (falha não escreve, ai/metadata/none, searchText/takenOn, access). Gate: `pnpm gate:fast` + int local.
3. **Fase 3 — Client + CLI + guardas + recibo** (quota ~0,5 dia): `archiveVisionApi.mjs`, `archiveCatalogPlan.mjs`, `catalog-archive-photos.mjs`, `assertLocalVisionEndpoint`, `pnpm archive:catalog`, pins de CI; unit do client/parser/guarda; plan/`--verify` contra o banco do worktree. Gate: `pnpm gate:fast`.
4. **Fase 4 — Busca/admin + entrega + lote** (quota ~0,5 dia): colunas/busca do admin, `.env.example`, `.gitignore`, runbook C232, changelog, plano no commit; `pnpm push`; CI high-risk (migration → curated e2e); deploy staging primeiro e canário `--limit` antes do lote completo (processamento ~4–10h de máquina, fora do appetite de código).
## Rabbit holes / Não escopo (engenharia)
- Embedding/vetor para busca semântica: guarda-se a descrição textual agora; a infra de C229 decide depois (gatilho abaixo).
- Reprocessamento por versão de prompt (`--refresh`) e refresh de metadados EXIF→GPS: adiados com gatilho.
- Fila/job in-app, retomada por página no CLI (a seleção por `catalogedAt` já é a retomada), paralelismo no engine.
- OCR dedicado além do VLM, saneamento/recorte de imagem, álbum público (C233) e biometria (C234).
- Fila/galeria de revisão própria: só se o gate pedir, com design nascido no gate.
- Segundo engine de catalogação, segundo cadastro de pessoa, provider `openai/*` no app (OPS123), qualquer superfície pública.
## Riscos e mitigação
1. **Qualidade do VLM local abaixo do aceite** — canário `--limit` no staging com amostra revisada por humano antes do lote; curadoria vence e `--refresh` fica adiado; gatilho de revisita reabre B/C de D1 com decisão explícita de PII.
2. **Engine fora do ar no meio do lote** — falha por foto é isolada e nomeada (HTTP/parse), nada é escrito, o lote não cai e a próxima execução retenta só as pendentes.
3. **Tamanho/tempo do lote** — 6,5k chamadas sequenciais (~2–5s cada) ≈ 4–10h; roda no container de manutenção com volume de recibos, `--limit` para canário e reexecução idempotente; o recibo fecha o tempo real.
4. **CI high-risk** — `src/collections/`, `src/migrations/` e `tests/helpers/` disparam unit/int full + e2e curado (`E2E_CURATED_SPECS`, `scripts/lib/e2e-affected-manifest.mjs:24-54`); `src/utilities/flickr` e o lib novo sem entry caem no fallback home smoke (`:396-404`); `src/utilities/municipality` idem; `src/utilities/access` não é tocado. Manter o PR focado para o orçamento de CI.
5. **Snapshot de migration** — C231 reparou a cadeia; ainda assim o diff gerado é revisado à mão contra o snapshot e aplicado no banco do worktree antes do PR; migrations já entregues nunca são editadas.
6. **Migração de schema não pré-aprovada** — apesar do `Status: aprovado` e do `--auto`, este plano inclui migração: o fluxo para para confirmação humana antes de executá-la; o deploy aplica migrations antes do build do runner (OPS66) e o alvo segue staging-primeiro.
7. **PII/rostos** — engine local + guarda de endpoint fail-closed (D2); o recibo não carrega imagem nem chave, só ids/contagens; nenhum caminho publica ou manda bytes para terceiro.
8. **`alt` sobrescrito** — só quando nunca curado; o alt do C231 é título/placeholder, então o refino é ganho; se a assessoria editar, o campo entra em `curatedFields` e nunca mais é tocado.
## Adiado com gatilho
- **Embedding/vetor** para busca semântica: gatilho = infra de C229 ou busca por termo não bastar.
- **`--refresh` por versão de prompt/engine**: gatilho = iteração de qualidade ou troca de engine.
- **EXIF→GPS para município**: gatilho = cobertura de geotag mostrar ganho material além do texto.
- **OCR dedicado**: gatilho = `visibleText` do VLM insuficiente para faixas/placas.
- **Fila/job in-app e re-catalogação sob demanda**: gatilho = necessidade real de reprocessar da ficha.
- **Álbum público (C233) e biometria (C234); superfície de revisão própria (Impeccable B)**: gatilho = gate.
## Aceite de engenharia
- [ ] Aceite de produto da intenção ainda coberto: ficha pré-catalogada revisável, busca interna por local/data/atividade/pessoa/termo, curadoria vence a IA, recibo honesto e idempotente, nenhum nome inventado, nada público.
- [ ] Invariantes AGENTS/engineering-standards: sem segundo engine/cadastro de pessoa, `openai/*` fora do app, `req` threaded em toda escrita multi-collection, `import 'server-only'` no módulo Payload, bypass com comentário, identificadores em inglês e copy pt-BR.
- [ ] Testes de domínio previstos (unit/int) onde write paths e curadoria mudam; sem e2e novo (OPS72) documentado; pins de CI atualizados (`SCRIPTS_SPEC_PINNED`, manifest).
- [ ] Migration aditiva gerada, revisada à mão, aplicada local e registrada; runbook C232, `.env.example`, `.gitignore` e changelog entregues; impl plan no commit.
- [ ] Guarda de localidade e confirmação de escrita provados por unit; `--apply` exige engine configurado e falha fechado sem `ARCHIVE_CATALOG_CONFIRM` fora de dev local.
- [ ] Gates: `pnpm gate:fast`, suíte int, e2e curado no CI, deploy staging-primeiro com canário antes do lote.
## Self-score (decision-quality)
1. Decisões caras com rejeitadas? **Sim** — D1–D8 têm opções, recomendação e rejeitadas explícitas.
2. Abordagem cabe no appetite? **Sim** — ~2 dias eng divididos em 4 fases de ~0,5 dia; o processamento é máquina, não código.
3. Rabbit holes nomeados? **Sim** — embedding, `--refresh`, GPS, OCR dedicado, fila, C233/C234, superfície própria.
4. Depth check: reusa shells/helpers? **Sim** — `downloadPrivateMediaToFile`, `withPayloadTransaction`, gazetteer/catálogos, `scripts/lib/cli.mjs`, esqueleto de CLI/recibo C231, padrão `curatedFields` C230.
5. Intenção permanece satisfeita? **Sim** — a engenharia escolheu módulos e engine sob os guardrails de produto, sem reescrever outcome.
**Score: 5/5.**

View file

@ -35,6 +35,7 @@
"media:recover": "cross-env NODE_OPTIONS=\"--no-deprecation --import=tsx/esm --import=./scripts/seed-loader.mjs\" node scripts/recover-media.mjs",
"reels:ingest": "cross-env NODE_OPTIONS=\"--no-deprecation --import=tsx/esm --import=./scripts/seed-loader.mjs\" node scripts/reels-ingest.mjs",
"flickr:import": "cross-env NODE_OPTIONS=\"--no-deprecation --import=tsx/esm --import=./scripts/seed-loader.mjs\" node scripts/ingest-flickr-archive.mjs",
"archive:catalog": "cross-env NODE_OPTIONS=\"--no-deprecation --import=tsx/esm --import=./scripts/seed-loader.mjs\" node scripts/catalog-archive-photos.mjs",
"db:seed:minimal": "cross-env NODE_OPTIONS=\"--no-deprecation --import=tsx/esm --import=./scripts/seed-loader.mjs\" node scripts/seed-minimal.mjs",
"campaign:staging:test-account": "cross-env NODE_OPTIONS=\"--no-deprecation --import=tsx/esm --import=./scripts/seed-loader.mjs\" node scripts/bootstrap-staging-test-account.mjs",
"db:check:push-chain": "node scripts/check-push-chain.mjs",

View file

@ -0,0 +1,284 @@
/**
* C232 — catalogues the private photo archive (C231) with the local vision
* engine: each pending `archivePhoto` is downloaded from the private storage,
* downscaled, analysed by an OpenAI-compatible VLM on the private network and
* written back as a revisable ficha — the assessoria's curation always wins and
* re-running converges ("processadas / puladas / falharam com motivo"). Nothing
* is published and no photo leaves the private network without the explicit
* remote escape; the model never names people.
*
* Modes:
* pnpm archive:catalog plan/dry-run (default) — lists the
* pending queue; no engine call, no write
* pnpm archive:catalog --apply catalogues the pending photos; requires
* ARCHIVE_CATALOG_CONFIRM=1 outside local
* dev, the complete S3_* envs and the
* vision engine envs
* pnpm archive:catalog --verify read-only inventory of the catalogue
*
* Options: --limit <n> (canary), --out <dir> (receipts, default `data/archive`),
* --help.
*
* Guards: --apply refuses a non-local/production target without the intent flag
* (C155) and demands the declared TEQO_ENV matching the exact database name
* (C195/C231); a remote target also demands the complete S3_* set; the vision
* endpoint must be on the private network unless ARCHIVE_VISION_ALLOW_REMOTE=1
* is declared (PII guardrail). The engine config lives in ARCHIVE_VISION_*
* (env only, never the repo).
*
* Runbook: on the homeserver, inside the compose maintenance service with the
* stack env file — see docs/ops/teqo-1313-deploy.md.
*/
import { mkdir, writeFile } from 'node:fs/promises'
import { dirname, join } from 'node:path'
import { getPayload } from 'payload'
import {
ARCHIVE_VISION_SYSTEM_PROMPT,
buildArchiveVisionPrompt,
parseArchiveVisionOutput,
} from '../src/lib/archivePhotoCatalog.ts'
import {
ARCHIVE_CATALOG_DEFAULT_OUT_DIR,
archiveCatalogReportStamp,
formatArchiveCatalogInventory,
formatArchiveCatalogReport,
parseArchiveCatalogCliArgs,
summarizeArchiveCatalogInventory,
summarizeArchiveCatalogResults,
} from './lib/archiveCatalogPlan.mjs'
import { analyzeArchivePhotoVision } from './lib/archiveVisionApi.mjs'
import {
TEQO_ENV_DATABASE_BY_ENV,
assertEnvironmentDatabaseTarget,
assertLocalVisionEndpoint,
assertWriteConfirm,
databaseTarget,
dieWithLabel,
isPrivateVisionHost,
loadCliEnv,
mirroredMediaRequired,
} from './lib/cli.mjs'
loadCliEnv()
const die = dieWithLabel('archive:catalog')
const WRITE_CONFIRM_FLAG = 'ARCHIVE_CATALOG_CONFIRM'
const envLines = Object.entries(TEQO_ENV_DATABASE_BY_ENV)
.map(([environment, database]) => `${environment} (${database})`)
.join(' | ')
const HELP = `
Uso: pnpm archive:catalog [opções]
Pré-cataloga as fotos do acervo privado (C231) com o engine de visão local: por
foto, legenda/descrição, atividade/cena, texto visível, temas do vocabulário
existente, município pelo gazetteer e pessoas públicas do catálogo curado.
A curadoria da assessoria vence: campo curado nunca é sobrescrito. Reexecutar
converge ("processadas / puladas / falharam com motivo") e nada é publicado.
Modos (mutuamente exclusivos; o default é o plano):
(sem flag) plano/dry-run — lista a fila pendente; sem engine nem escrita
--apply cataloga as fotos pendentes (engine local + escrita transacional)
--verify inventário read-only da catalogação (sem engine)
Opções:
--limit <n> --apply: processa no máximo n fotos (canário); no plano, só limita a prévia
--out <dir> diretório dos recibos (default ${ARCHIVE_CATALOG_DEFAULT_OUT_DIR})
--help esta ajuda
Ambiente:
ARCHIVE_VISION_BASE_URL endpoint OpenAI-compatible do engine (ex.: http://100.94.122.26:11434/v1)
ARCHIVE_VISION_MODEL modelo de visão (ex.: qwen2.5vl:7b)
ARCHIVE_VISION_API_KEY opcional, quando o servidor local exige chave
DATABASE_URL alvo (obrigatório em todos os modos)
--apply em alvo não-local/produção exige ${WRITE_CONFIRM_FLAG}=1, TEQO_ENV=${envLines}
casando o banco exato, as quatro envs S3_* e o engine na rede privada (host
público exige ARCHIVE_VISION_ALLOW_REMOTE=1). O plano e o --verify são read-only.
`
/** Renders the configured engine without calling it (plan/verify banner). */
const configuredEngine = () => {
const raw = String(process.env.ARCHIVE_VISION_BASE_URL ?? '').trim()
if (raw === '') return null
try {
const host = new URL(raw).hostname
return {
host,
scope: isPrivateVisionHost(host) ? 'local' : 'remote',
model: String(process.env.ARCHIVE_VISION_MODEL ?? '').trim() || '(não definido)',
}
} catch {
return null
}
}
const writeReport = async (options, report, suffix = '') => {
const stamp = archiveCatalogReportStamp(report.runAt)
const reportPath = join(options.out, 'reports', `archive-catalog-${stamp}${suffix}.json`)
await mkdir(dirname(reportPath), { recursive: true })
await writeFile(reportPath, JSON.stringify(report, null, 2))
return reportPath
}
async function main() {
const options = parseArchiveCatalogCliArgs(process.argv.slice(2))
if (options.help) {
console.log(HELP)
process.exit(0)
}
if (!process.env.DATABASE_URL) die('DATABASE_URL não definida; recusando continuar.')
const mode = options.apply ? 'apply' : options.verify ? 'verify' : 'plan'
const runAt = new Date().toISOString()
console.log(`[archive:catalog] alvo: ${databaseTarget()} | modo: ${mode}`)
let engine = configuredEngine()
if (mode === 'apply') {
assertWriteConfirm({
label: 'archive:catalog',
flag: WRITE_CONFIRM_FLAG,
command: 'pnpm archive:catalog --apply',
})
assertEnvironmentDatabaseTarget()
const { resolveS3StorageEnv } = await import('../src/utilities/mediaStorage.ts')
const storage = resolveS3StorageEnv(process.env)
if (mirroredMediaRequired({ s3Enabled: storage.enabled })) {
die(
'escrita em produção/remoto exige mídia espelhada: configure S3_BUCKET, S3_ENDPOINT, S3_ACCESS_KEY_ID e S3_SECRET_ACCESS_KEY; sem elas o acervo não está acessível no alvo.',
)
}
const { scope, host } = assertLocalVisionEndpoint()
const model = String(process.env.ARCHIVE_VISION_MODEL ?? '').trim()
if (model === '') {
die('ARCHIVE_VISION_MODEL ausente — informe o modelo de visão (ex.: qwen2.5vl:7b).')
}
engine = { host, scope, model }
}
const config = (await import('../src/payload.config.ts')).default
const { listArchivePhotos } = await import('../src/utilities/flickr/archivePhotoIngest.ts')
const { catalogArchivePhoto, listArchivePhotoCatalogQueue } =
await import('../src/utilities/flickr/archivePhotoCatalog.ts')
const payload = await getPayload({ config })
const startedAt = Date.now()
if (mode === 'verify') {
const rows = await listArchivePhotos(payload)
const inventory = summarizeArchiveCatalogInventory(rows)
const report = {
runAt,
mode,
target: databaseTarget(),
engine,
...inventory,
durationMs: Date.now() - startedAt,
}
const reportPath = await writeReport(options, report, '-verify')
for (const line of formatArchiveCatalogInventory(inventory)) console.log(line)
console.log(`\n[archive:catalog] recibo JSON: ${reportPath}`)
if (inventory.missingFilename.length > 0) {
die(
`${inventory.missingFilename.length} foto(s) sem arquivo armazenado — acervo inconsistente.`,
)
}
if (inventory.pending > 0) {
die(`${inventory.pending} foto(s) ainda pendentes de catalogação — rode o --apply.`)
}
process.exit(0)
}
if (mode === 'plan') {
// The plan always reports the TRUE pending count; `--limit` only caps the
// preview (the apply mode is where it caps the work).
const pending = await listArchivePhotoCatalogQueue({ payload })
const report = {
runAt,
mode,
target: databaseTarget(),
engine,
pending: pending.length,
photos: pending.slice(0, options.limit ?? 50).map((item) => ({
flickrId: item.flickrId,
title: item.title,
takenAt: item.takenAt,
})),
durationMs: Date.now() - startedAt,
}
const reportPath = await writeReport(options, report)
for (const line of formatArchiveCatalogReport(report)) console.log(line)
console.log(`\n[archive:catalog] recibo JSON: ${reportPath}`)
process.exit(0)
}
const queue = await listArchivePhotoCatalogQueue({ payload, limit: options.limit })
console.log(`[archive:catalog] fila: ${queue.length} foto(s) pendente(s)`)
/**
* The engine seam: one HTTP call + the fail-closed parse against the closed
* vocabularies. A transport error or an off-contract answer throws and the
* pipeline turns it into a named failure — never a write.
*/
const analyze = async ({ imageDataUrl }) => {
const content = await analyzeArchivePhotoVision({
baseUrl: process.env.ARCHIVE_VISION_BASE_URL,
model: process.env.ARCHIVE_VISION_MODEL,
apiKey: process.env.ARCHIVE_VISION_API_KEY,
imageDataUrl,
prompt: buildArchiveVisionPrompt(),
systemPrompt: ARCHIVE_VISION_SYSTEM_PROMPT,
})
const suggestion = parseArchiveVisionOutput(content)
if (!suggestion) {
throw new Error('resposta do engine fora do contrato JSON da catalogação')
}
return suggestion
}
const results = []
for (const item of queue) {
const result = await catalogArchivePhoto({ payload, item, analyze })
results.push({ flickrId: item.flickrId, ...result })
const detail =
result.status === 'cataloged'
? ` (${result.source})`
: result.status === 'failed'
? ` [${result.stage}]: ${result.error}`
: ''
console.log(`[archive:catalog] ${result.status}${detail} ${item.flickrId}`)
}
const summary = summarizeArchiveCatalogResults(results)
const report = {
runAt,
mode,
target: databaseTarget(),
engine,
summary,
results: results.map((result) => ({
flickrId: result.flickrId,
status: result.status,
source: result.source ?? null,
stage: result.stage ?? null,
error: result.error ?? null,
})),
durationMs: Date.now() - startedAt,
}
const reportPath = await writeReport(options, report)
for (const line of formatArchiveCatalogReport(report)) console.log(line)
console.log(`\n[archive:catalog] recibo JSON: ${reportPath}`)
if (summary.cataloged === 0 && results.length > 0) {
die('nenhuma foto foi catalogada — falhas listadas no recibo.')
}
process.exit(0)
}
main().catch((error) => {
die(error?.message || String(error))
})

View file

@ -0,0 +1,195 @@
/**
* C232 — pure planning/reporting of the photo-archive cataloguing: the argv
* parser, the run summaries (by outcome and by source), the catalogue
* `--verify` inventory and the human lines + JSON receipt. No I/O of its own:
* the queue, the engine and the Payload writes arrive as parameters, so the
* unit spec drives everything with fakes.
*/
const USAGE = 'uso: `pnpm archive:catalog [--apply|--verify] [--limit <n>] [--out <dir>]`'
export const ARCHIVE_CATALOG_DEFAULT_OUT_DIR = 'data/archive'
export const archiveCatalogReportStamp = (runAt) => String(runAt).replace(/[:.]/g, '-')
/**
* `pnpm archive:catalog` argv parser — pure, so the unit spec pins it (the CLI
* owns the exit and the help text). The default mode is the plan/dry-run;
* errors carry the usage line.
*
* @param {string[]} [argv]
* @returns {{ apply: boolean, verify: boolean, limit: number | null, out: string, help: boolean }}
*/
export const parseArchiveCatalogCliArgs = (argv = process.argv.slice(2)) => {
const options = {
apply: false,
verify: false,
limit: null,
out: ARCHIVE_CATALOG_DEFAULT_OUT_DIR,
help: false,
}
for (let index = 0; index < argv.length; index += 1) {
const arg = argv[index]
const value = () => {
const next = argv[++index]
if (next === undefined || next.startsWith('--')) {
throw new Error(`faltou valor para ${arg} — ${USAGE}.`)
}
return next
}
if (arg === '--help' || arg === '-h') options.help = true
else if (arg === '--apply') options.apply = true
else if (arg === '--verify') options.verify = true
else if (arg === '--limit') options.limit = Number(value())
else if (arg === '--out') options.out = value()
else throw new Error(`argumento desconhecido: ${arg} — ${USAGE}.`)
}
if (options.help) return options
if (options.apply && options.verify) {
throw new Error(`--apply e --verify são mutuamente exclusivos — ${USAGE}.`)
}
if (options.limit !== null && (!Number.isInteger(options.limit) || options.limit < 1)) {
throw new Error('--limit deve ser >= 1.')
}
if (options.out.split(/[\\/]/).includes('..')) {
throw new Error('--out não pode escapar do diretório do repo (sem "..").')
}
return options
}
const ARCHIVE_CATALOG_SOURCE_KEYS = ['ai', 'metadata', 'none']
const emptyBySource = () => ({ ai: 0, metadata: 0, none: 0 })
/**
* Aggregates one run: what was catalogued (and from where), what was skipped
* (catalogued before the run) and what failed with stage+reason.
*
* @param {Array<{ flickrId: string, status: string, source?: string | null, stage?: string | null, error?: string | null }>} results
*/
export const summarizeArchiveCatalogResults = (results) => {
const bySource = emptyBySource()
const failures = []
let cataloged = 0
let skipped = 0
let failed = 0
for (const result of results) {
if (result.status === 'cataloged') {
cataloged += 1
if (ARCHIVE_CATALOG_SOURCE_KEYS.includes(result.source)) bySource[result.source] += 1
} else if (result.status === 'skipped') {
skipped += 1
} else {
failed += 1
failures.push({
flickrId: result.flickrId,
stage: result.stage ?? null,
error: result.error ?? null,
})
}
}
return { cataloged, skipped, failed, bySource, failures }
}
/**
* The `--verify` inventory over the stored rows (read-only): coverage of the
* catalogue by source, the pending count (the honest gap this check fails on,
* mirroring the `flickr:import --verify` missing-file rule) and the rows
* without a stored filename (the one broken state C231 already checks).
*
* @param {any[]} rows
*/
export const summarizeArchiveCatalogInventory = (rows) => {
const bySource = emptyBySource()
const missingFilename = []
let cataloged = 0
let pending = 0
let withoutScene = 0
let withoutMunicipality = 0
for (const row of rows) {
if (!row?.filename) missingFilename.push(String(row?.flickrId ?? ''))
const catalog = row?.catalog ?? {}
if (!catalog.catalogedAt) {
pending += 1
continue
}
cataloged += 1
if (ARCHIVE_CATALOG_SOURCE_KEYS.includes(catalog.source)) bySource[catalog.source] += 1
if (!catalog.scene) withoutScene += 1
if (catalog.municipality === undefined || catalog.municipality === null)
withoutMunicipality += 1
}
return {
total: rows.length,
cataloged,
pending,
bySource,
withoutScene,
withoutMunicipality,
missingFilename,
}
}
const REPORT_LABEL = '[archive:catalog]'
const engineLine = (engine) =>
engine ? `${REPORT_LABEL} engine: ${engine.host} (${engine.model}, ${engine.scope})` : null
/**
* Human lines for stdout; the JSON file keeps the full report (the CLI already
* announced target/mode before booting Payload). Failures are never hidden —
* each one names its stage and reason.
*
* @param {Record<string, any>} report
* @returns {string[]}
*/
export const formatArchiveCatalogReport = (report) => {
const label = REPORT_LABEL
const lines = []
const engine = engineLine(report.engine)
if (engine) lines.push(engine)
if (report.mode === 'plan') {
lines.push(`${label} pendentes na fila: ${report.pending}`)
}
if (report.summary) {
const { cataloged, skipped, failed, bySource } = report.summary
lines.push(
`${label} resultado: processadas: ${cataloged} · puladas (já catalogadas): ${skipped} · falharam: ${failed}`,
)
lines.push(
`${label} origem: IA: ${bySource.ai} · metadados: ${bySource.metadata} · nada a propor: ${bySource.none}`,
)
}
lines.push(`${label} tempo: ${(report.durationMs / 1000).toFixed(1)}s`)
for (const failure of report.summary?.failures ?? []) {
lines.push(
` - ${failure.flickrId} (${failure.stage ?? 'desconhecido'}): ${failure.error ?? 'erro'}`,
)
}
return lines
}
/** `--verify` human lines (catalogue inventory, read-only). */
export const formatArchiveCatalogInventory = (inventory) => {
const label = REPORT_LABEL
const lines = [
`${label} inventário: ${inventory.total} foto(s) · catalogadas: ${inventory.cataloged} · pendentes: ${inventory.pending}`,
`${label} origem: IA: ${inventory.bySource.ai} · metadados: ${inventory.bySource.metadata} · nada a propor: ${inventory.bySource.none}`,
`${label} cobertura: sem cena: ${inventory.withoutScene} · sem município: ${inventory.withoutMunicipality}`,
]
if (inventory.missingFilename.length > 0) {
lines.push(`${label} sem arquivo armazenado: ${inventory.missingFilename.join(', ')}`)
}
if (inventory.pending > 0) {
lines.push(`${label} ainda pendentes de catalogação: ${inventory.pending}`)
}
return lines
}

View file

@ -0,0 +1,145 @@
/**
* C232 — thin OpenAI-compatible vision client for the photo-archive
* cataloguing: one POST to `{base}/chat/completions` with the image as a
* `data:` URL, plain `fetch` (injectable), timeout and retry with backoff for
* transient failures only. There is no SDK dependency and no vendor pinned:
* any local server speaking the protocol (Ollama, llama.cpp, vLLM) fits.
*
* The API key, when the server has one, travels in the `Authorization` header
* and never enters logs or receipts. A 4xx answer is terminal (a wrong model
* name must not be retried); 429/5xx/network/timeout retry.
*/
const DEFAULT_TIMEOUT_MS = 90_000
const DEFAULT_RETRY_ATTEMPTS = 3
const DEFAULT_RETRY_BASE_DELAY_MS = 2_000
const DEFAULT_MAX_OUTPUT_TOKENS = 900
const DEFAULT_TEMPERATURE = 0.2
export class ArchiveVisionError extends Error {
/**
* @param {string} message
* @param {{ status?: number | null }} [options]
*/
constructor(message, { status = null } = {}) {
super(message)
this.name = 'ArchiveVisionError'
this.status = status
}
}
/** HTTP 429/5xx, aborts and network errors retry; a 4xx refusal never does. */
const isRetryable = (error) => {
if (error instanceof ArchiveVisionError) {
return error.status === 429 || (error.status !== null && error.status >= 500)
}
return true
}
/**
* One vision call returning the raw assistant content (the caller parses it
* against the closed vocabularies). `imageDataUrl` is the prepared JPEG as a
* `data:image/jpeg;base64,…` string; the original never leaves the machine
* that prepared it.
*
* @param {{
* baseUrl?: string | null,
* model?: string | null,
* apiKey?: string | null,
* imageDataUrl: string,
* prompt: string,
* systemPrompt: string,
* fetchImpl?: typeof fetch,
* timeoutMs?: number,
* retryAttempts?: number,
* retryBaseDelayMs?: number,
* maxOutputTokens?: number,
* temperature?: number,
* sleep?: (ms: number) => Promise<void>,
* }} options
* @returns {Promise<string>}
*/
export const analyzeArchivePhotoVision = async ({
baseUrl,
model,
apiKey = '',
imageDataUrl,
prompt,
systemPrompt,
fetchImpl = fetch,
timeoutMs = DEFAULT_TIMEOUT_MS,
retryAttempts = DEFAULT_RETRY_ATTEMPTS,
retryBaseDelayMs = DEFAULT_RETRY_BASE_DELAY_MS,
maxOutputTokens = DEFAULT_MAX_OUTPUT_TOKENS,
temperature = DEFAULT_TEMPERATURE,
sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms)),
}) => {
const base = String(baseUrl ?? '')
.trim()
.replace(/\/+$/, '')
if (base === '') throw new ArchiveVisionError('ARCHIVE_VISION_BASE_URL ausente.')
const chosenModel = String(model ?? '').trim()
if (chosenModel === '') throw new ArchiveVisionError('ARCHIVE_VISION_MODEL ausente.')
if (typeof imageDataUrl !== 'string' || !imageDataUrl.startsWith('data:image/')) {
throw new ArchiveVisionError('imagem preparada inválida (esperado data: URL).')
}
const headers = { 'Content-Type': 'application/json', Accept: 'application/json' }
const key = String(apiKey ?? '').trim()
if (key !== '') headers.Authorization = `Bearer ${key}`
const body = JSON.stringify({
model: chosenModel,
temperature,
max_tokens: maxOutputTokens,
stream: false,
messages: [
{ role: 'system', content: systemPrompt },
{
role: 'user',
content: [
{ type: 'text', text: prompt },
{ type: 'image_url', image_url: { url: imageDataUrl } },
],
},
],
})
for (let attempt = 1; ; attempt += 1) {
try {
const response = await fetchImpl(`${base}/chat/completions`, {
method: 'POST',
headers,
body,
signal: AbortSignal.timeout(timeoutMs),
})
if (!response.ok) {
// The provider's error body is operational gold ("model not found") —
// clipped into the receipt message; never the request, image or key.
let detail = ''
try {
detail = String(await response.text())
.replace(/\s+/g, ' ')
.trim()
.slice(0, 200)
} catch {
detail = ''
}
throw new ArchiveVisionError(`HTTP ${response.status}${detail ? `: ${detail}` : ''}`, {
status: response.status,
})
}
const parsed = await response.json().catch(() => {
throw new ArchiveVisionError('resposta do engine não é JSON.')
})
const content = parsed?.choices?.[0]?.message?.content
if (typeof content !== 'string' || content.trim() === '') {
throw new ArchiveVisionError('resposta do engine sem conteúdo de texto.')
}
return content
} catch (error) {
if (attempt >= retryAttempts || !isRetryable(error)) throw error
await sleep(retryBaseDelayMs * 2 ** (attempt - 1))
}
}
}

View file

@ -319,6 +319,89 @@ export const assertWriteConfirm = ({ label, flag, command, hint = '' }) => {
)
}
/**
* True when the host is reachable only from the private network: loopback,
* RFC1918, CGNAT/tailnet (`100.64/10`), link-local/ULA IPv6, `.local`/`.internal`
* suffixes or a single-label name (docker network). A public IP or domain is
* NOT private — it is exactly what the C232 guard refuses by default.
*
* @param {unknown} hostname
* @returns {boolean}
*/
export const isPrivateVisionHost = (hostname) => {
const host = String(hostname ?? '')
.trim()
.toLowerCase()
.replace(/^\[|\]$/g, '')
if (host === '') return false
// IPv6 literal (loopback/link-local/ULA). The `includes(':')` gate is what
// keeps a NAME starting with "fc"/"fd" (e.g. fdic.gov) from passing here —
// it must fall through to the domain checks below.
if (
host.includes(':') &&
(host === '::1' || host.startsWith('fe80:') || host.startsWith('fc') || host.startsWith('fd'))
) {
return true
}
if (!host.includes('.')) return true
if (host.endsWith('.local') || host.endsWith('.internal')) return true
const parts = host.split('.')
if (parts.length !== 4 || parts.some((part) => !/^\d{1,3}$/.test(part))) return false
const octets = parts.map(Number)
if (octets.some((octet) => octet > 255)) return false
const [first, second] = octets
return (
first === 10 ||
first === 127 ||
(first === 192 && second === 168) ||
(first === 172 && second >= 16 && second <= 31) ||
(first === 100 && second >= 64 && second <= 127)
)
}
/**
* C232 — the vision engine reads the citizens' photos, so it must live on the
* private network: a public host is refused unless the run declares the
* `ARCHIVE_VISION_ALLOW_REMOTE=1` escape (the product guardrail is "PII/rostos
* não vão a terceiro sem decisão explícita e contrato"). Returns the scope so
* the receipt records which mode ran — the escape is never invisible.
*
* @param {{ baseUrl?: string | null, allowRemote?: boolean, label?: string }} [options]
* @returns {{ scope: 'local' | 'remote', host: string }}
*/
export const assertLocalVisionEndpoint = ({
baseUrl = process.env.ARCHIVE_VISION_BASE_URL,
allowRemote = isTruthyEnv(process.env.ARCHIVE_VISION_ALLOW_REMOTE),
label = 'archive:catalog',
} = {}) => {
const raw = String(baseUrl ?? '').trim()
if (raw === '') {
dieWithLabel(label)(
'ARCHIVE_VISION_BASE_URL ausente — configure o endpoint do engine de visão (ex.: http://100.94.122.26:11434/v1).',
)
}
let url
try {
url = new URL(raw)
} catch {
dieWithLabel(label)(`ARCHIVE_VISION_BASE_URL inválida: ${raw}`)
}
if (url.protocol !== 'http:' && url.protocol !== 'https:') {
dieWithLabel(label)(`ARCHIVE_VISION_BASE_URL deve ser http(s), não "${url.protocol}".`)
}
const host = url.hostname
if (isPrivateVisionHost(host)) return { scope: 'local', host }
if (allowRemote) return { scope: 'remote', host }
dieWithLabel(label)(
`ARCHIVE_VISION_BASE_URL aponta para host público "${host}" — as fotos do acervo (PII/rostos de cidadãos) não vão a terceiro sem decisão explícita e contrato.\n` +
' Rode o engine na rede privada (LAN/tailnet) ou, ciente do risco, declare ARCHIVE_VISION_ALLOW_REMOTE=1.',
)
}
/**
* Port for `next dev` (OPS40). Next's CLI resolves its port via commander's
* `.env('PORT')` BEFORE `@next/env` loads `.env.local`, so a `PORT` written by

View file

@ -43,6 +43,8 @@ export const SCRIPTS_SPEC_PINNED = [
'scripts/lib/agent-pool-state.mjs',
'scripts/lib/agent-session.mjs',
'scripts/lib/ansi.mjs',
'scripts/lib/archiveCatalogPlan.mjs',
'scripts/lib/archiveVisionApi.mjs',
'scripts/lib/auto-unblock.mjs',
'scripts/lib/briefingContent.mjs',
'scripts/lib/briefingRender.mjs',

View file

@ -1,6 +1,20 @@
import type { CollectionConfig } from 'payload'
import type { CollectionBeforeValidateHook, CollectionConfig, Field } from 'payload'
import { ARCHIVE_PHOTO_SLUG } from '@/lib/archivePhoto'
import {
ARCHIVE_PHOTO_CAPTION_MAX_LENGTH,
ARCHIVE_PHOTO_CATALOG_SOURCES,
ARCHIVE_PHOTO_CURATED_FIELDS,
ARCHIVE_PHOTO_DESCRIPTION_MAX_LENGTH,
ARCHIVE_PHOTO_SCENES,
ARCHIVE_PHOTO_VISIBLE_TEXT_MAX_LENGTH,
archivePhotoCatalogSourceLabels,
archivePhotoCuratedFieldLabels,
archivePhotoSearchText,
archivePhotoTakenOn,
changedArchivePhotoCuratedFields,
} from '@/lib/archivePhotoCatalog'
import { SPEECH_TOPICS } from '@/lib/speechFacets'
import { canReadArchivePhoto, payloadAdminOnly } from '@/utilities/campaignAccess'
/**
@ -16,8 +30,186 @@ import { canReadArchivePhoto, payloadAdminOnly } from '@/utilities/campaignAcces
* (`archivePhotoAlt`); the remaining Flickr fields document provenance and feed
* the catalogue. The stored name is deterministic (`flickr-<id>.<ext>`), so a
* re-run converges by overwriting the same object key.
*
* C232 adds the curated pre-cataloguing on these same rows: the `catalog` group
* (proposal + honest `source`/`catalogedAt` state), `curatedFields` (what the
* assessoria edited — the pipeline never overwrites it), `takenOn` (date facet
* derived from the Flickr wall clock) and `searchText` (the normalized haystack
* the admin list search matches).
*/
const catalogGroup: Field = {
name: 'catalog',
type: 'group',
label: 'Pré-catalogação',
admin: {
description:
'Proposta automática (IA + metadados). A assessoria revisa na ficha: o que ela editar entra em "Campos curados" e nunca é sobrescrito.',
},
fields: [
{
name: 'caption',
type: 'text',
label: 'Legenda',
maxLength: ARCHIVE_PHOTO_CAPTION_MAX_LENGTH,
},
{
name: 'description',
type: 'textarea',
label: 'Descrição',
maxLength: ARCHIVE_PHOTO_DESCRIPTION_MAX_LENGTH,
admin: {
description: 'Descrição semântica do que a foto mostra (não substitui a do Flickr).',
},
},
{
name: 'scene',
type: 'select',
label: 'Atividade/cena',
index: true,
options: ARCHIVE_PHOTO_SCENES.map((scene) => ({ value: scene.value, label: scene.label })),
},
{
name: 'visibleText',
type: 'textarea',
label: 'Texto visível',
maxLength: ARCHIVE_PHOTO_VISIBLE_TEXT_MAX_LENGTH,
admin: {
description: 'Texto legível na imagem (faixas, placas, banners).',
},
},
{
name: 'hasPeople',
type: 'checkbox',
label: 'Pessoas na cena',
admin: {
description: 'Presença de pessoas na foto — sem identificação de rosto (biometria é C234).',
},
},
{
name: 'themes',
type: 'select',
label: 'Temas',
hasMany: true,
index: true,
options: SPEECH_TOPICS.map((topic) => ({ value: topic.value, label: topic.label })),
},
{
name: 'people',
type: 'text',
hasMany: true,
label: 'Pessoas públicas',
admin: {
description:
'Somente nomes do catálogo curado, mencionados no texto da foto; nunca inferidos de rosto.',
},
},
{
name: 'municipality',
type: 'relationship',
relationTo: 'municipality',
label: 'Município',
index: true,
admin: {
description: 'Quando o texto da foto aponta um município sem ambiguidade.',
},
},
{
name: 'source',
type: 'select',
label: 'Origem',
options: ARCHIVE_PHOTO_CATALOG_SOURCES.map((source) => ({
value: source,
label: archivePhotoCatalogSourceLabels[source],
})),
admin: {
readOnly: true,
description: 'IA = o modelo contribuiu; Metadados = só o texto derivou; Nada a propor.',
},
},
{
name: 'catalogedAt',
type: 'date',
label: 'Catalogada em',
index: true,
admin: {
readOnly: true,
description: 'Chave de idempotência: foto com data preenchida não é reprocessada.',
},
},
],
}
/**
* C232 — the catalogue index the admin list uses: `takenOn` (date facet derived
* from the Flickr wall clock) and `searchText` (normalized haystack). When a
* request carries a user (an admin edit), the fields that actually changed are
* added to `curatedFields` — the cataloguing/hook never marks on system writes
* (`context.archivePhotoCatalog`, or no user at all).
*/
const deriveArchivePhotoCatalogIndex: CollectionBeforeValidateHook = async ({
data,
originalDoc,
req,
}) => {
if (!data) return data
if (req.user && !req.context?.archivePhotoCatalog) {
const changed = changedArchivePhotoCuratedFields({ data, originalDoc })
if (changed.length > 0) {
const curated = new Set<string>(originalDoc?.curatedFields ?? [])
for (const field of changed) curated.add(field)
data.curatedFields = [...curated]
}
}
const takenAt = data.takenAt ?? originalDoc?.takenAt ?? null
data.takenOn = archivePhotoTakenOn(takenAt)
const catalog = { ...(originalDoc?.catalog ?? {}), ...(data.catalog ?? {}) }
const rawMunicipality = catalog.municipality
const municipalityId =
typeof rawMunicipality === 'number'
? rawMunicipality
: rawMunicipality && typeof rawMunicipality === 'object'
? rawMunicipality.id
: null
const municipality =
municipalityId != null
? await req.payload
.findByID({
collection: 'municipality',
id: municipalityId,
depth: 0,
select: { name: true },
// Intentional admin bypass: the município catalog is read-only geography.
overrideAccess: true,
req,
})
.catch(() => null)
: null
data.searchText = archivePhotoSearchText({
alt: data.alt ?? originalDoc?.alt,
title: data.title ?? originalDoc?.title,
description: data.description ?? originalDoc?.description,
catalogDescription: catalog.description,
caption: catalog.caption,
scene: catalog.scene,
visibleText: catalog.visibleText,
themes: catalog.themes,
people: catalog.people,
municipalityName: municipality?.name ?? null,
albumTitles: (data.albums ?? originalDoc?.albums ?? []).map(
(album: { title?: string | null }) => album?.title,
),
tagNames: (data.tags ?? originalDoc?.tags ?? []).map(
(tag: { name?: string | null }) => tag?.name,
),
})
return data
}
export const ArchivePhoto: CollectionConfig = {
slug: ARCHIVE_PHOTO_SLUG,
labels: {
@ -28,6 +220,9 @@ export const ArchivePhoto: CollectionConfig = {
group: 'Comunicação',
description:
'Originais e metadados do acervo de fotos do Flickr (conta depjorgesolla). Só abrem com login da campanha.',
useAsTitle: 'alt',
defaultColumns: ['alt', 'takenOn', 'catalog.scene', 'catalog.municipality', 'catalog.source'],
listSearchableFields: ['searchText'],
},
access: {
create: payloadAdminOnly,
@ -35,6 +230,9 @@ export const ArchivePhoto: CollectionConfig = {
update: payloadAdminOnly,
delete: payloadAdminOnly,
},
hooks: {
beforeValidate: [deriveArchivePhotoCatalogIndex],
},
fields: [
{
name: 'flickrId',
@ -57,6 +255,22 @@ export const ArchivePhoto: CollectionConfig = {
description: 'Descrição do arquivo para acessibilidade.',
},
},
catalogGroup,
{
name: 'curatedFields',
type: 'select',
label: 'Campos curados pela assessoria',
hasMany: true,
options: ARCHIVE_PHOTO_CURATED_FIELDS.map((value) => ({
value,
label: archivePhotoCuratedFieldLabels[value],
})),
admin: {
readOnly: true,
description:
'O que a assessoria já editou; a catalogação automática nunca sobrescreve estes campos.',
},
},
{
name: 'title',
type: 'text',
@ -89,6 +303,16 @@ export const ArchivePhoto: CollectionConfig = {
description: 'Data informada pelo Flickr, sem fuso (YYYY-MM-DD HH:MM:SS).',
},
},
{
name: 'takenOn',
type: 'date',
label: 'Dia da foto',
index: true,
admin: {
readOnly: true,
description: 'Derivado da data da foto (meio-dia UTC, sem fuso).',
},
},
{
name: 'postedAt',
type: 'text',
@ -157,6 +381,16 @@ export const ArchivePhoto: CollectionConfig = {
type: 'text',
label: 'Licença (Flickr)',
},
{
name: 'searchText',
type: 'textarea',
label: 'Texto normalizado (busca)',
admin: {
readOnly: true,
description:
'Tudo o que a busca da lista encontra, normalizado (sem acentos, minúsculas): legenda, texto visível, temas, pessoas, município, álbuns e tags.',
},
},
],
upload: true,
}

View file

@ -0,0 +1,506 @@
/**
* C232 — pure contract of the AI pre-cataloguing of the photo archive: the
* closed scene vocabulary, the curated-field vocabulary (the same mechanism
* C230 persists on a content piece), the source states of a ficha, the vision
* output parser and the deterministic merge that decides what gets written.
* No I/O and no `server-only`: the collection hooks, the Payload pipeline, the
* CLI and the tests share this module.
*
* The rules encoded here are product rules: the model never names people (only
* the curated `publicFigureCatalog` matched against TEXT does, and the
* assessoria confirms on the ficha), the scene/themes are closed vocabularies
* (unknown tokens are dropped, never invented) and a failed call is not a
* state — it stays out of the row so the next run retries it.
*/
import { z } from 'zod'
import { publicFigureCatalog } from '@/lib/publicFigureCatalog'
import { SPEECH_TOPICS, resolveSpeechTopic, type SpeechTopic } from '@/lib/speechFacets'
import { escapeRegExp, normalizeForSearch } from '@/lib/speechSearch'
/** Closed vocabulary of atividade/cena (the facet C233 consumes). */
export const ARCHIVE_PHOTO_SCENES = [
{ value: 'plenaria', label: 'Plenária' },
{ value: 'audiencia', label: 'Audiência' },
{ value: 'reuniao', label: 'Reunião' },
{ value: 'evento', label: 'Evento' },
{ value: 'mobilizacao', label: 'Mobilização' },
{ value: 'visita', label: 'Visita' },
{ value: 'entrevista', label: 'Entrevista' },
{ value: 'discurso', label: 'Discurso' },
{ value: 'retrato', label: 'Retrato' },
{ value: 'gabinete', label: 'Gabinete' },
{ value: 'outro', label: 'Outro' },
] as const
export type ArchivePhotoScene = (typeof ARCHIVE_PHOTO_SCENES)[number]['value']
/**
* Resolves a scene token (value or pt-BR label, accent/case insensitive) to its
* taxonomy entry. Fail-closed: unknown tokens return null.
*/
export const resolveArchivePhotoScene = (token: unknown): ArchivePhotoScene | null => {
if (typeof token !== 'string') return null
const key = normalizeForSearch(token).replace(/-/g, ' ')
if (!key) return null
const match = ARCHIVE_PHOTO_SCENES.find(
(scene) =>
normalizeForSearch(scene.value).replace(/-/g, ' ') === key ||
normalizeForSearch(scene.label) === key,
)
return match?.value ?? null
}
const archivePhotoSceneLabel = (value: ArchivePhotoScene): string =>
ARCHIVE_PHOTO_SCENES.find((scene) => scene.value === value)?.label ?? value
/**
* The fields a human edit freezes: the cataloguing never writes a field listed
* in `curatedFields` (C230's mechanism, one literal per persisted field name).
*/
export const ARCHIVE_PHOTO_CURATED_FIELDS = [
'alt',
'caption',
'description',
'scene',
'visibleText',
'hasPeople',
'themes',
'people',
'municipality',
] as const
export type ArchivePhotoCuratedField = (typeof ARCHIVE_PHOTO_CURATED_FIELDS)[number]
/** pt-BR names of the curated fields for the admin's read-only chip list. */
export const archivePhotoCuratedFieldLabels: Record<ArchivePhotoCuratedField, string> = {
alt: 'Texto alternativo',
caption: 'Legenda',
description: 'Descrição',
scene: 'Atividade/cena',
visibleText: 'Texto visível',
hasPeople: 'Pessoas na cena',
themes: 'Temas',
people: 'Pessoas públicas',
municipality: 'Município',
}
/** Outcome of one photo: the model contributed, only metadata did, or nothing. */
export const ARCHIVE_PHOTO_CATALOG_SOURCES = ['ai', 'metadata', 'none'] as const
export type ArchivePhotoCatalogSource = (typeof ARCHIVE_PHOTO_CATALOG_SOURCES)[number]
export const archivePhotoCatalogSourceLabels: Record<ArchivePhotoCatalogSource, string> = {
ai: 'IA',
metadata: 'Metadados',
none: 'Nada a propor',
}
export const ARCHIVE_PHOTO_CAPTION_MAX_LENGTH = 200
export const ARCHIVE_PHOTO_DESCRIPTION_MAX_LENGTH = 600
export const ARCHIVE_PHOTO_VISIBLE_TEXT_MAX_LENGTH = 1000
const ARCHIVE_PHOTO_ALT_MAX_LENGTH = 300
const ARCHIVE_PHOTO_PEOPLE_MAX = 8
/** What one vision call proposes; every field can be empty (nothing to propose). */
export type ArchivePhotoSuggestion = {
caption: string | null
description: string | null
scene: ArchivePhotoScene | null
visibleText: string | null
hasPeople: boolean | null
themes: SpeechTopic[]
}
const clip = (value: string | null, maximum: number): string | null => {
if (value === null) return null
const text = value.trim().slice(0, maximum).trim()
return text === '' ? null : text
}
/**
* Resolves the model's theme tokens against the existing taxonomy (value or
* pt-BR label); unknown tokens are dropped and duplicates collapse. Never
* invents a slug outside `SPEECH_TOPICS`.
*/
export const archivePhotoThemesFrom = (values: readonly unknown[]): SpeechTopic[] => {
const themes: SpeechTopic[] = []
for (const value of values) {
const entry = typeof value === 'string' ? resolveSpeechTopic(value) : null
if (entry && !themes.includes(entry.value)) themes.push(entry.value)
}
return themes
}
/**
* Names of the curated catalog mentioned in the TEXT (title, album, caption,
* visible text…). This is a reading of what is written — never of a face — and
* an unresolved or ambiguous value never becomes a name. Whole-word,
* accent/case insensitive, same normalization family as the catalog resolver.
*/
export const matchPublicFigureMentions = (text: string): string[] => {
const haystack = normalizeForSearch(text)
if (!haystack) return []
const mentioned: string[] = []
for (const entry of publicFigureCatalog) {
const needle = normalizeForSearch(entry.name)
if (!needle) continue
const pattern = new RegExp(`(^|[^a-z0-9])${escapeRegExp(needle)}([^a-z0-9]|$)`)
if (!pattern.test(haystack)) continue
if (!mentioned.includes(entry.name) && mentioned.length < ARCHIVE_PHOTO_PEOPLE_MAX) {
mentioned.push(entry.name)
}
}
return mentioned
}
const archiveVisionField = {
text: z.string().nullish().catch(null),
boolean: z.boolean().nullish().catch(null),
}
const archiveVisionOutputSchema = z
.object({
caption: archiveVisionField.text,
description: archiveVisionField.text,
scene: archiveVisionField.text,
visibleText: archiveVisionField.text,
hasPeople: archiveVisionField.boolean,
themes: z.array(z.unknown()).nullish().catch(null),
})
.passthrough()
const parseJsonObject = (raw: string): unknown => {
const text = raw
.trim()
.replace(/^```(?:json)?/i, '')
.replace(/```$/, '')
.trim()
try {
return JSON.parse(text)
} catch {
return null
}
}
/**
* Fail-closed parser of one vision answer: a non-object (or unparsable) answer
* is null — the caller reports a named failure and retries next run. Inside a
* valid object each field degrades alone: an invalid scene or theme is
* dropped, never guessed.
*/
export const parseArchiveVisionOutput = (raw: unknown): ArchivePhotoSuggestion | null => {
const parsed = typeof raw === 'string' ? parseJsonObject(raw) : raw
if (parsed === null || typeof parsed !== 'object' || Array.isArray(parsed)) return null
const result = archiveVisionOutputSchema.safeParse(parsed)
if (!result.success) return null
const data = result.data
return {
caption: clip(data.caption ?? null, ARCHIVE_PHOTO_CAPTION_MAX_LENGTH),
description: clip(data.description ?? null, ARCHIVE_PHOTO_DESCRIPTION_MAX_LENGTH),
scene: resolveArchivePhotoScene(data.scene),
visibleText: clip(data.visibleText ?? null, ARCHIVE_PHOTO_VISIBLE_TEXT_MAX_LENGTH),
hasPeople: data.hasPeople ?? null,
themes: archivePhotoThemesFrom(data.themes ?? []),
}
}
type ArchivePhotoSearchInput = {
alt?: string | null
title?: string | null
description?: string | null
catalogDescription?: string | null
caption?: string | null
scene?: ArchivePhotoScene | null
visibleText?: string | null
themes?: readonly SpeechTopic[] | null
people?: readonly string[] | null
municipalityName?: string | null
albumTitles?: readonly string[] | null
tagNames?: readonly string[] | null
}
/**
* The normalized haystack the admin list search matches (the same shape C199/
* C230 persist): what a person would type to find the photo — names, places,
* themes, what is written in it.
*/
export const archivePhotoSearchText = (input: ArchivePhotoSearchInput): string =>
normalizeForSearch(
[
input.alt ?? '',
input.title ?? '',
input.description ?? '',
input.catalogDescription ?? '',
input.caption ?? '',
input.scene ? archivePhotoSceneLabel(input.scene) : '',
input.visibleText ?? '',
...(input.themes ?? []).map((theme) => resolveSpeechTopic(theme)?.label ?? theme),
...(input.people ?? []),
input.municipalityName ?? '',
...(input.albumTitles ?? []),
...(input.tagNames ?? []),
]
.map((value) => value.trim())
.filter(Boolean)
.join(' '),
)
/**
* The date facet derived from the Flickr wall clock (`YYYY-MM-DD HH:MM:SS`):
* fixed at noon UTC so the admin never shows the previous day. Invalid input
* (or an impossible date) is null — never a guessed day.
*/
export const archivePhotoTakenOn = (takenAt: unknown): string | null => {
if (typeof takenAt !== 'string') return null
const match = /^(\d{4})-(\d{2})-(\d{2})(?:[ T]|$)/.exec(takenAt.trim())
if (!match) return null
const [, year, month, day] = match
const utc = Date.UTC(Number(year), Number(month) - 1, Number(day), 12)
const date = new Date(utc)
if (
date.getUTCFullYear() !== Number(year) ||
date.getUTCMonth() !== Number(month) - 1 ||
date.getUTCDate() !== Number(day)
) {
return null
}
return date.toISOString()
}
const comparableCuratedValue = (field: ArchivePhotoCuratedField, value: unknown): unknown => {
if (field === 'municipality') {
if (typeof value === 'number') return value
if (value !== null && typeof value === 'object' && 'id' in (value as object)) {
return (value as { id: unknown }).id
}
return null
}
if (field === 'themes' || field === 'people') {
return Array.isArray(value)
? value.filter((entry): entry is string => typeof entry === 'string' && entry.trim() !== '')
: []
}
if (field === 'hasPeople') {
return typeof value === 'boolean' ? value : null
}
return typeof value === 'string' ? value : null
}
/**
* The admin edit path marks which fields the assessoria touched: an incoming
* value that differs from the stored one (arrays normalized, clearing counts)
* enters `curatedFields`. Only runs for a request with a user — the system
* writes (ingest, cataloguing) never mark.
*/
export const changedArchivePhotoCuratedFields = ({
data,
originalDoc,
}: {
data: Record<string, unknown> | null | undefined
originalDoc: Record<string, unknown> | null | undefined
}): ArchivePhotoCuratedField[] => {
if (!data) return []
const catalog = data.catalog
const incomingCatalog =
catalog !== null && typeof catalog === 'object' ? (catalog as Record<string, unknown>) : null
const originalCatalog = originalDoc?.catalog
const storedCatalog =
originalCatalog !== null && typeof originalCatalog === 'object'
? (originalCatalog as Record<string, unknown>)
: null
const changed: ArchivePhotoCuratedField[] = []
for (const field of ARCHIVE_PHOTO_CURATED_FIELDS) {
const incoming = field === 'alt' ? data.alt : incomingCatalog?.[field]
if (incoming === undefined) continue
const stored = field === 'alt' ? originalDoc?.alt : storedCatalog?.[field]
const before = comparableCuratedValue(field, stored)
const after = comparableCuratedValue(field, incoming)
if (JSON.stringify(before) !== JSON.stringify(after)) changed.push(field)
}
return changed
}
type ArchivePhotoCatalogValues = {
caption?: string
description?: string
scene?: ArchivePhotoScene
visibleText?: string
hasPeople?: boolean
themes?: SpeechTopic[]
people?: string[]
municipality?: number
source: ArchivePhotoCatalogSource
catalogedAt: string
}
export type ArchivePhotoCatalogWrite = {
data: {
alt?: string
catalog: ArchivePhotoCatalogValues
}
source: ArchivePhotoCatalogSource
}
/**
* The deterministic merge of one photo: proposed values (model + text
* metadata) fill only fields the assessoria never curated, and the source
* states what actually landed — `ai` when a model field was written,
* `metadata` when only the text-derived ones were, `none` when there was
* nothing to propose. `catalogedAt` is set on every outcome, so "nada a
* propor" is a final honest state instead of a silent retry loop.
*/
export const buildArchivePhotoCatalogWrite = ({
suggestion,
municipalityId,
mentionedPeople,
gazetteerThemes,
curatedFields,
currentCatalog,
currentAlt,
catalogedAt,
}: {
suggestion: ArchivePhotoSuggestion | null
municipalityId: number | null
mentionedPeople: readonly string[]
gazetteerThemes: readonly SpeechTopic[]
curatedFields: readonly string[]
currentCatalog?: {
caption?: string | null
description?: string | null
scene?: ArchivePhotoScene | null
visibleText?: string | null
hasPeople?: boolean | null
themes?: SpeechTopic[] | null
people?: string[] | null
municipality?: number | null
} | null
currentAlt?: string | null
catalogedAt: string
}): ArchivePhotoCatalogWrite => {
const curated = new Set<string>(curatedFields)
const current: NonNullable<typeof currentCatalog> = currentCatalog ?? {}
/** A proposed value that will be written: not curated, not blank/empty. */
const writes = (field: ArchivePhotoCuratedField, value: unknown): boolean =>
!curated.has(field) &&
value !== null &&
value !== undefined &&
!(typeof value === 'string' && value.trim() === '') &&
!(Array.isArray(value) && value.length === 0)
/**
* Kept curated value when the field is frozen (verbatim), otherwise the
* proposal normalized: blank strings and empty arrays are "nothing to
* propose", never a written value.
*/
const pick = <Key extends keyof typeof current>(
field: Key,
proposed: NonNullable<(typeof current)[Key]> | null | undefined,
): NonNullable<(typeof current)[Key]> | undefined => {
if (curated.has(field)) {
const kept = current[field]
return kept === null || kept === undefined
? undefined
: (kept as NonNullable<(typeof current)[Key]>)
}
if (proposed === null || proposed === undefined) return undefined
if (Array.isArray(proposed)) {
return proposed.length === 0 ? undefined : (proposed as NonNullable<(typeof current)[Key]>)
}
if (typeof proposed === 'string') {
const text = proposed.trim()
return text === '' ? undefined : (text as NonNullable<(typeof current)[Key]>)
}
return proposed as NonNullable<(typeof current)[Key]>
}
const aiThemes = suggestion?.themes ?? []
const themes = [...new Set<SpeechTopic>([...aiThemes, ...gazetteerThemes])]
const people = [...new Set(mentionedPeople)]
const caption = suggestion?.caption ?? null
const written = {
caption: pick('caption', caption),
description: pick('description', suggestion?.description ?? null),
scene: pick('scene', suggestion?.scene ?? null),
visibleText: pick('visibleText', suggestion?.visibleText ?? null),
hasPeople: pick('hasPeople', suggestion?.hasPeople ?? null),
themes: pick('themes', themes.length > 0 ? themes : null),
people: pick('people', people.length > 0 ? people : null),
municipality: pick('municipality', municipalityId),
}
const modelWrote =
writes('caption', caption) ||
writes('description', suggestion?.description) ||
writes('scene', suggestion?.scene) ||
writes('visibleText', suggestion?.visibleText) ||
writes('hasPeople', suggestion?.hasPeople) ||
writes('themes', aiThemes)
const metadataWrote =
!modelWrote &&
(writes('municipality', municipalityId) ||
writes('people', people) ||
writes('themes', gazetteerThemes))
const source: ArchivePhotoCatalogSource = modelWrote ? 'ai' : metadataWrote ? 'metadata' : 'none'
// The AI refines the alt only from its OWN caption (never from human text):
// a curated caption leaves the alt exactly where it was.
const alt = curated.has('alt')
? (currentAlt ?? undefined)
: curated.has('caption')
? undefined
: (clip(caption, ARCHIVE_PHOTO_ALT_MAX_LENGTH) ?? undefined)
return {
data: {
...(alt !== undefined ? { alt } : {}),
catalog: {
...written,
source,
catalogedAt,
},
},
source,
}
}
const ARCHIVE_VISION_SCENE_LINES = ARCHIVE_PHOTO_SCENES.filter((scene) => scene.value !== 'outro')
.map((scene) => `${scene.value}=${scene.label}`)
.join('; ')
/** The theme vocabulary is offered from its owner — no copied literal to drift. */
const ARCHIVE_VISION_THEME_LINES = SPEECH_TOPICS.map(
({ value, label }) => `${value}=${label}`,
).join('; ')
export const ARCHIVE_VISION_SYSTEM_PROMPT =
'Você pré-cataloga fotos do acervo do deputado federal Jorge Solla (PT-BA) para uma base interna da campanha. ' +
'Descreva apenas o que está visível na imagem: nunca invente lugar, data, evento, número ou fala. ' +
'NUNCA identifique pessoas por rosto nem escreva nomes de pessoas (nem do deputado). ' +
'Não escreva nomes de cidades: o local é resolvido por outra via. ' +
'Responda SOMENTE com um objeto JSON válido, sem markdown e sem comentários.'
/**
* The user prompt of one photo: the closed vocabularies the answer must use.
* The model is free to answer null/empty for anything it cannot see — the
* parser drops whatever leaves the vocabulary.
*/
export const buildArchiveVisionPrompt = (): string =>
[
'Analise a foto e devolva um JSON com exatamente estas chaves:',
'{"caption": "legenda curta em português (máx. 200 caracteres)",',
' "description": "uma ou duas frases factuais em português (máx. 600 caracteres)",',
' "scene": "um dos valores de cena ou null",',
' "visibleText": "texto legível de faixas, placas e banners ou null",',
' "hasPeople": true, false ou null,',
' "themes": ["zero ou mais valores de tema"]}',
'',
`Cenas: ${ARCHIVE_VISION_SCENE_LINES}; outro=Outro (use outro só se nenhuma se aplicar).`,
`Temas: ${ARCHIVE_VISION_THEME_LINES}.`,
'Use null ou lista vazia quando não houver certeza; é melhor não propor do que inventar.',
].join('\n')

View file

@ -6,7 +6,7 @@ import {
type SpeechScope,
type SpeechTopic,
} from '@/lib/speechFacets'
import { normalizeForSearch } from '@/lib/speechSearch'
import { escapeRegExp, normalizeForSearch } from '@/lib/speechSearch'
/**
* Offline gazetteer pass of the speech facet classifier (C153): topic/scope
@ -287,8 +287,6 @@ const SPEECH_SCOPE_TERMS: Record<SpeechScope, readonly string[]> = {
],
}
const escapeRegExp = (value: string): string => value.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')
const containsTerm = (normalizedText: string, term: string): boolean =>
new RegExp(`(^|[^a-z0-9])${escapeRegExp(term)}($|[^a-z0-9])`).test(normalizedText)

View file

@ -16,6 +16,9 @@ export const normalizeForSearch = (value: string): string =>
.replace(/\s+/g, ' ')
.trim()
/** Escapes a literal for a RegExp source; the whole-word matchers share it. */
export const escapeRegExp = (value: string): string => value.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')
type SpeechSearchableRecord = {
/** Normalized segment text persisted by the segment hook (C153). */
searchText?: string | null

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,88 @@
import { MigrateUpArgs, MigrateDownArgs, sql } from '@payloadcms/db-postgres'
export async function up({ db, payload, req }: MigrateUpArgs): Promise<void> {
await db.execute(sql`
CREATE EXTENSION IF NOT EXISTS pg_trgm;
CREATE TYPE "public"."enum_archive_photo_catalog_themes" AS ENUM('saude', 'educacao', 'cultura', 'esporte', 'seguranca-publica', 'meio-ambiente', 'economia-trabalho', 'direitos-humanos', 'infraestrutura', 'ciencia-tecnologia', 'politica-instituicoes', 'agricultura', 'habitacao-cidades', 'comunicacao-midia', 'igualdade-racial', 'mulheres-genero', 'juventude', 'pessoa-deficiencia');
CREATE TYPE "public"."enum_archive_photo_curated_fields" AS ENUM('alt', 'caption', 'description', 'scene', 'visibleText', 'hasPeople', 'themes', 'people', 'municipality');
CREATE TYPE "public"."enum_archive_photo_catalog_scene" AS ENUM('plenaria', 'audiencia', 'reuniao', 'evento', 'mobilizacao', 'visita', 'entrevista', 'discurso', 'retrato', 'gabinete', 'outro');
CREATE TYPE "public"."enum_archive_photo_catalog_source" AS ENUM('ai', 'metadata', 'none');
CREATE TABLE "archive_photo_catalog_themes" (
"order" integer NOT NULL,
"parent_id" integer NOT NULL,
"value" "enum_archive_photo_catalog_themes",
"id" serial PRIMARY KEY NOT NULL
);
CREATE TABLE "archive_photo_curated_fields" (
"order" integer NOT NULL,
"parent_id" integer NOT NULL,
"value" "enum_archive_photo_curated_fields",
"id" serial PRIMARY KEY NOT NULL
);
CREATE TABLE "archive_photo_texts" (
"id" serial PRIMARY KEY NOT NULL,
"order" integer NOT NULL,
"parent_id" integer NOT NULL,
"path" varchar NOT NULL,
"text" varchar
);
ALTER TABLE "archive_photo" ADD COLUMN "catalog_caption" varchar;
ALTER TABLE "archive_photo" ADD COLUMN "catalog_description" varchar;
ALTER TABLE "archive_photo" ADD COLUMN "catalog_scene" "enum_archive_photo_catalog_scene";
ALTER TABLE "archive_photo" ADD COLUMN "catalog_visible_text" varchar;
ALTER TABLE "archive_photo" ADD COLUMN "catalog_has_people" boolean;
ALTER TABLE "archive_photo" ADD COLUMN "catalog_municipality_id" integer;
ALTER TABLE "archive_photo" ADD COLUMN "catalog_source" "enum_archive_photo_catalog_source";
ALTER TABLE "archive_photo" ADD COLUMN "catalog_cataloged_at" timestamp(3) with time zone;
ALTER TABLE "archive_photo" ADD COLUMN "taken_on" timestamp(3) with time zone;
ALTER TABLE "archive_photo" ADD COLUMN "search_text" varchar;
ALTER TABLE "archive_photo_catalog_themes" ADD CONSTRAINT "archive_photo_catalog_themes_parent_fk" FOREIGN KEY ("parent_id") REFERENCES "public"."archive_photo"("id") ON DELETE cascade ON UPDATE no action;
ALTER TABLE "archive_photo_curated_fields" ADD CONSTRAINT "archive_photo_curated_fields_parent_fk" FOREIGN KEY ("parent_id") REFERENCES "public"."archive_photo"("id") ON DELETE cascade ON UPDATE no action;
ALTER TABLE "archive_photo_texts" ADD CONSTRAINT "archive_photo_texts_parent_fk" FOREIGN KEY ("parent_id") REFERENCES "public"."archive_photo"("id") ON DELETE cascade ON UPDATE no action;
CREATE INDEX "archive_photo_catalog_themes_order_idx" ON "archive_photo_catalog_themes" USING btree ("order");
CREATE INDEX "archive_photo_catalog_themes_parent_idx" ON "archive_photo_catalog_themes" USING btree ("parent_id");
CREATE INDEX "archive_photo_catalog_themes_value_idx" ON "archive_photo_catalog_themes" USING btree ("value");
CREATE INDEX "archive_photo_curated_fields_order_idx" ON "archive_photo_curated_fields" USING btree ("order");
CREATE INDEX "archive_photo_curated_fields_parent_idx" ON "archive_photo_curated_fields" USING btree ("parent_id");
CREATE INDEX "archive_photo_texts_order_parent" ON "archive_photo_texts" USING btree ("order","parent_id");
ALTER TABLE "archive_photo" ADD CONSTRAINT "archive_photo_catalog_municipality_id_municipality_id_fk" FOREIGN KEY ("catalog_municipality_id") REFERENCES "public"."municipality"("id") ON DELETE set null ON UPDATE no action;
CREATE INDEX "archive_photo_catalog_catalog_scene_idx" ON "archive_photo" USING btree ("catalog_scene");
CREATE INDEX "archive_photo_catalog_catalog_municipality_idx" ON "archive_photo" USING btree ("catalog_municipality_id");
CREATE INDEX "archive_photo_catalog_catalog_cataloged_at_idx" ON "archive_photo" USING btree ("catalog_cataloged_at");
CREATE INDEX "archive_photo_taken_on_idx" ON "archive_photo" USING btree ("taken_on");
CREATE INDEX IF NOT EXISTS "archive_photo_search_text_trgm_idx" ON "archive_photo" USING gin ("search_text" gin_trgm_ops);`)
}
export async function down({ db, payload, req }: MigrateDownArgs): Promise<void> {
await db.execute(sql`
DROP INDEX IF EXISTS "archive_photo_search_text_trgm_idx";
ALTER TABLE "archive_photo_catalog_themes" DISABLE ROW LEVEL SECURITY;
ALTER TABLE "archive_photo_curated_fields" DISABLE ROW LEVEL SECURITY;
ALTER TABLE "archive_photo_texts" DISABLE ROW LEVEL SECURITY;
DROP TABLE "archive_photo_catalog_themes" CASCADE;
DROP TABLE "archive_photo_curated_fields" CASCADE;
DROP TABLE "archive_photo_texts" CASCADE;
ALTER TABLE "archive_photo" DROP CONSTRAINT "archive_photo_catalog_municipality_id_municipality_id_fk";
DROP INDEX "archive_photo_catalog_catalog_scene_idx";
DROP INDEX "archive_photo_catalog_catalog_municipality_idx";
DROP INDEX "archive_photo_catalog_catalog_cataloged_at_idx";
DROP INDEX "archive_photo_taken_on_idx";
ALTER TABLE "archive_photo" DROP COLUMN "catalog_caption";
ALTER TABLE "archive_photo" DROP COLUMN "catalog_description";
ALTER TABLE "archive_photo" DROP COLUMN "catalog_scene";
ALTER TABLE "archive_photo" DROP COLUMN "catalog_visible_text";
ALTER TABLE "archive_photo" DROP COLUMN "catalog_has_people";
ALTER TABLE "archive_photo" DROP COLUMN "catalog_municipality_id";
ALTER TABLE "archive_photo" DROP COLUMN "catalog_source";
ALTER TABLE "archive_photo" DROP COLUMN "catalog_cataloged_at";
ALTER TABLE "archive_photo" DROP COLUMN "taken_on";
ALTER TABLE "archive_photo" DROP COLUMN "search_text";
DROP TYPE "public"."enum_archive_photo_catalog_themes";
DROP TYPE "public"."enum_archive_photo_curated_fields";
DROP TYPE "public"."enum_archive_photo_catalog_scene";
DROP TYPE "public"."enum_archive_photo_catalog_source";`)
}

View file

@ -88,6 +88,7 @@ import * as migration_20260924_045625_add_content_piece_link_failure_reason from
import * as migration_20260924_123940_add_content_piece_people from './20260924_123940_add_content_piece_people';
import * as migration_20260926_034118_add_speech_embedding from './20260926_034118_add_speech_embedding';
import * as migration_20260926_062039_add_archive_photo from './20260926_062039_add_archive_photo';
import * as migration_20260926_132106_add_archive_photo_catalog from './20260926_132106_add_archive_photo_catalog';
export const migrations = [
{
@ -533,11 +534,16 @@ export const migrations = [
{
up: migration_20260926_034118_add_speech_embedding.up,
down: migration_20260926_034118_add_speech_embedding.down,
name: '20260926_034118_add_speech_embedding'
name: '20260926_034118_add_speech_embedding',
},
{
up: migration_20260926_062039_add_archive_photo.up,
down: migration_20260926_062039_add_archive_photo.down,
name: '20260926_062039_add_archive_photo'
name: '20260926_062039_add_archive_photo',
},
{
up: migration_20260926_132106_add_archive_photo_catalog.up,
down: migration_20260926_132106_add_archive_photo_catalog.down,
name: '20260926_132106_add_archive_photo_catalog',
},
];

View file

@ -1412,6 +1412,93 @@ export interface ArchivePhoto {
* Descrição do arquivo para acessibilidade.
*/
alt: string;
/**
* Proposta automática (IA + metadados). A assessoria revisa na ficha: o que ela editar entra em "Campos curados" e nunca é sobrescrito.
*/
catalog?: {
caption?: string | null;
/**
* Descrição semântica do que a foto mostra (não substitui a do Flickr).
*/
description?: string | null;
scene?:
| (
| 'plenaria'
| 'audiencia'
| 'reuniao'
| 'evento'
| 'mobilizacao'
| 'visita'
| 'entrevista'
| 'discurso'
| 'retrato'
| 'gabinete'
| 'outro'
)
| null;
/**
* Texto legível na imagem (faixas, placas, banners).
*/
visibleText?: string | null;
/**
* Presença de pessoas na foto — sem identificação de rosto (biometria é C234).
*/
hasPeople?: boolean | null;
themes?:
| (
| 'saude'
| 'educacao'
| 'cultura'
| 'esporte'
| 'seguranca-publica'
| 'meio-ambiente'
| 'economia-trabalho'
| 'direitos-humanos'
| 'infraestrutura'
| 'ciencia-tecnologia'
| 'politica-instituicoes'
| 'agricultura'
| 'habitacao-cidades'
| 'comunicacao-midia'
| 'igualdade-racial'
| 'mulheres-genero'
| 'juventude'
| 'pessoa-deficiencia'
)[]
| null;
/**
* Somente nomes do catálogo curado, mencionados no texto da foto; nunca inferidos de rosto.
*/
people?: string[] | null;
/**
* Quando o texto da foto aponta um município sem ambiguidade.
*/
municipality?: (number | null) | Municipality;
/**
* IA = o modelo contribuiu; Metadados = só o texto derivou; Nada a propor.
*/
source?: ('ai' | 'metadata' | 'none') | null;
/**
* Chave de idempotência: foto com data preenchida não é reprocessada.
*/
catalogedAt?: string | null;
};
/**
* O que a assessoria já editou; a catalogação automática nunca sobrescreve estes campos.
*/
curatedFields?:
| (
| 'alt'
| 'caption'
| 'description'
| 'scene'
| 'visibleText'
| 'hasPeople'
| 'themes'
| 'people'
| 'municipality'
)[]
| null;
title?: string | null;
description?: string | null;
tags?:
@ -1424,6 +1511,10 @@ export interface ArchivePhoto {
* Data informada pelo Flickr, sem fuso (YYYY-MM-DD HH:MM:SS).
*/
takenAt?: string | null;
/**
* Derivado da data da foto (meio-dia UTC, sem fuso).
*/
takenOn?: string | null;
/**
* Data de publicação no Flickr (ISO 8601, UTC).
*/
@ -1454,6 +1545,10 @@ export interface ArchivePhoto {
sourceUrl?: string | null;
owner?: string | null;
license?: string | null;
/**
* Tudo o que a busca da lista encontra, normalizado (sem acentos, minúsculas): legenda, texto visível, temas, pessoas, município, álbuns e tags.
*/
searchText?: string | null;
updatedAt: string;
createdAt: string;
url?: string | null;
@ -3045,6 +3140,21 @@ export interface ContentMediaSelect<T extends boolean = true> {
export interface ArchivePhotoSelect<T extends boolean = true> {
flickrId?: T;
alt?: T;
catalog?:
| T
| {
caption?: T;
description?: T;
scene?: T;
visibleText?: T;
hasPeople?: T;
themes?: T;
people?: T;
municipality?: T;
source?: T;
catalogedAt?: T;
};
curatedFields?: T;
title?: T;
description?: T;
tags?:
@ -3054,6 +3164,7 @@ export interface ArchivePhotoSelect<T extends boolean = true> {
id?: T;
};
takenAt?: T;
takenOn?: T;
postedAt?: T;
albums?:
| T
@ -3072,6 +3183,7 @@ export interface ArchivePhotoSelect<T extends boolean = true> {
sourceUrl?: T;
owner?: T;
license?: T;
searchText?: T;
updatedAt?: T;
createdAt?: T;
url?: T;

View file

@ -15,12 +15,13 @@ import {
} from '@/lib/contentPiece'
import { institutionCatalog, institutionSpellings } from '@/lib/institutionCatalog'
import { type SpeechTopic } from '@/lib/speechFacets'
import { matchMunicipalityMentions, type SpeechFacetInput } from '@/lib/speechGazetteer'
import { normalizeForSearch } from '@/lib/speechSearch'
import { type SpeechFacetInput } from '@/lib/speechGazetteer'
import { escapeRegExp, normalizeForSearch } from '@/lib/speechSearch'
import {
DEEPSEEK_FLASH_STRUCTURED_MAX_OUTPUT_TOKENS,
DEEPSEEK_FLASH_STRUCTURED_PROVIDER_OPTIONS,
} from '@/utilities/ai/deepseekFlashDefaults'
import { resolveMentionedMunicipalityId } from '@/utilities/municipality/municipalityMentionResolver'
import { classifySpeech } from '@/utilities/speech/speechClassifier'
/**
@ -76,8 +77,6 @@ const SYSTEM_PROMPT =
const clip = (value: string, maximum: number): string => value.trim().slice(0, maximum).trim()
const escapeRegExp = (value: string): string => value.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')
/** Deterministic fallback: never empty, never invents a fact. */
export const buildContentPieceFallbackMetadata = ({
type,
@ -151,10 +150,10 @@ const suggestContentPieceMetadata: ContentPieceSuggester = async ({
}
/**
* The city rule is deliberately conservative: the gazetteer mention list is
* converted to distinct cities, and only a city with exactly ONE catalog entry
* resolves. Salvador has one entry per TSE zone, so a bare "Salvador" mention
* is ambiguous and stays empty — the assessoria picks the zone on the ficha.
* The content-piece entry point to the shared conservative city rule
* (`resolveMentionedMunicipalityId`): only a city with exactly ONE catalog
* entry resolves, and a bare "Salvador" stays empty — the assessoria picks the
* zone on the ficha.
*/
export const resolveContentPieceMunicipalityId = async ({
payload,
@ -162,24 +161,7 @@ export const resolveContentPieceMunicipalityId = async ({
}: {
payload: Payload
transcript: string
}): Promise<number | null> => {
const mentions = matchMunicipalityMentions(transcript)
const cities = new Set(mentions.map((entry) => entry.city))
if (cities.size !== 1) return null
const entries = mentions.filter((entry) => entry.city === [...cities][0])
if (entries.length !== 1) return null
const found = await payload.find({
collection: 'municipality',
where: { slug: { equals: entries[0]!.slug } },
depth: 0,
limit: 1,
pagination: false,
// Intentional admin bypass: the município catalog is read-only geography.
overrideAccess: true,
})
return found.docs[0]?.id ?? null
}
}): Promise<number | null> => resolveMentionedMunicipalityId({ payload, text: transcript })
/**
* Matches the institution catalog against the text: a spelling counts only as a

View file

@ -0,0 +1,277 @@
import 'server-only'
import { mkdtemp, rm } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import type { Payload } from 'payload'
import sharp from 'sharp'
import { ARCHIVE_PHOTO_SLUG } from '@/lib/archivePhoto'
import {
buildArchivePhotoCatalogWrite,
matchPublicFigureMentions,
type ArchivePhotoCatalogSource,
type ArchivePhotoSuggestion,
} from '@/lib/archivePhotoCatalog'
import { classifySpeechByGazetteer } from '@/lib/speechGazetteer'
import { messageOf } from '@/utilities/media/ffmpeg'
import { resolveMentionedMunicipalityId } from '@/utilities/municipality/municipalityMentionResolver'
import { withPayloadTransaction } from '@/utilities/payloadTransaction'
import {
downloadPrivateMediaToFile,
resolvePrivateMediaStaticDir,
} from '@/utilities/privateMedia/privateMediaResponse'
/**
* C232 — Payload-side cataloguing of one archive photo: the private bytes are
* downloaded to a temp file, downscaled for the engine, answered by the
* injected analyzer and merged with the deterministic text rules (gazetteer
* municipality/themes, curated-catalog people) inside ONE transaction that
* re-reads `curatedFields` and never overwrites a human decision. The engine
* is a seam: `src/` never imports `scripts/`; the CLI injects the HTTP client.
*
* Failure of one photo is returned, never thrown — the caller owns the receipt,
* nothing is written and the next run retries it (a failure is NOT a state).
* The model never names people and there is no biometrics here.
*/
/** Long edge sent to the engine; the original never leaves the archive. */
const IMAGE_MAX_EDGE = 1024
const IMAGE_JPEG_QUALITY = 80
export type ArchivePhotoCatalogItem = {
id: number
flickrId: string
filename: string | null
mimeType: string | null
title: string | null
description: string | null
takenAt: string | null
alt: string
albums: { title: string }[]
tags: string[]
}
export type ArchivePhotoAnalyzer = (input: {
imageDataUrl: string
}) => Promise<ArchivePhotoSuggestion>
export type ArchivePhotoCatalogResult =
| { status: 'cataloged'; source: ArchivePhotoCatalogSource }
| { status: 'skipped' }
| {
status: 'failed'
stage: 'download' | 'image' | 'analyze' | 'resolve' | 'write'
error: string
}
const toItem = (doc: {
id: number
flickrId: string
filename?: string | null
mimeType?: string | null
title?: string | null
description?: string | null
takenAt?: string | null
alt: string
albums?: ({ title?: string | null } | null)[] | null
tags?: ({ name?: string | null } | null)[] | null
}): ArchivePhotoCatalogItem => ({
id: doc.id,
flickrId: doc.flickrId,
filename: doc.filename ?? null,
mimeType: doc.mimeType ?? null,
title: doc.title ?? null,
description: doc.description ?? null,
takenAt: doc.takenAt ?? null,
alt: doc.alt,
albums: (doc.albums ?? [])
.map((album) => album?.title ?? '')
.filter((title) => title !== '')
.map((title) => ({ title })),
tags: (doc.tags ?? []).map((tag) => tag?.name ?? '').filter((name) => name !== ''),
})
const catalogSelect = {
flickrId: true,
filename: true,
mimeType: true,
title: true,
description: true,
takenAt: true,
alt: true,
albums: true,
tags: true,
} as const
/**
* The photos still waiting for a catalogue, oldest first. `catalogedAt` is the
* idempotency key: a catalogued photo (any source, including "nada a propor")
* never enters the queue again. `limit` is the canary cap — when absent, the
* whole queue is paged read-only.
*/
export const listArchivePhotoCatalogQueue = async ({
payload,
limit,
}: {
payload: Payload
limit?: number | null
}): Promise<ArchivePhotoCatalogItem[]> => {
const items: ArchivePhotoCatalogItem[] = []
let page = 1
let totalPages = 1
do {
const found = await payload.find({
collection: ARCHIVE_PHOTO_SLUG,
where: { 'catalog.catalogedAt': { exists: false } },
page,
limit: 100,
depth: 0,
sort: 'id',
select: catalogSelect,
// Intentional bypass: the catalogue CLI is a trusted actor with no session.
overrideAccess: true,
})
for (const doc of found.docs) {
items.push(toItem(doc))
if (limit != null && items.length >= limit) return items
}
totalPages = found.totalPages
page += 1
} while (page <= totalPages)
return items
}
/**
* One photo through the whole pipeline. Phases fail with a NAMED stage so the
* receipt says why: download/image never reach the engine, analyze is the
* provider, write is the transaction. `skipped` means the photo was catalogued
* after the queue was read (fresh read inside the transaction).
*/
export const catalogArchivePhoto = async ({
payload,
item,
analyze,
}: {
payload: Payload
item: ArchivePhotoCatalogItem
analyze: ArchivePhotoAnalyzer
}): Promise<ArchivePhotoCatalogResult> => {
let tempDir: string | null = null
try {
tempDir = await mkdtemp(join(tmpdir(), 'archive-photo-'))
const originalPath = join(tempDir, 'original')
try {
await downloadPrivateMediaToFile({
media: { filename: item.filename, mimeType: item.mimeType },
staticDir: resolvePrivateMediaStaticDir(payload, ARCHIVE_PHOTO_SLUG),
destinationPath: originalPath,
})
} catch (error) {
return { status: 'failed', stage: 'download', error: messageOf(error, 'erro desconhecido') }
}
let imageDataUrl: string
try {
const resized = await sharp(originalPath)
.rotate()
.resize({
width: IMAGE_MAX_EDGE,
height: IMAGE_MAX_EDGE,
fit: 'inside',
withoutEnlargement: true,
})
.jpeg({ quality: IMAGE_JPEG_QUALITY })
.toBuffer()
imageDataUrl = `data:image/jpeg;base64,${resized.toString('base64')}`
} catch (error) {
return { status: 'failed', stage: 'image', error: messageOf(error, 'erro desconhecido') }
}
let suggestion: ArchivePhotoSuggestion
try {
suggestion = await analyze({ imageDataUrl })
} catch (error) {
return { status: 'failed', stage: 'analyze', error: messageOf(error, 'erro desconhecido') }
}
const text = [
item.alt,
item.title,
item.description,
...item.albums.map((album) => album.title),
...item.tags,
suggestion.caption,
suggestion.visibleText,
]
.filter((value): value is string => typeof value === 'string' && value.trim() !== '')
.join(' ')
let municipalityId: number | null
try {
municipalityId = await resolveMentionedMunicipalityId({ payload, text })
} catch (error) {
return { status: 'failed', stage: 'resolve', error: messageOf(error, 'erro desconhecido') }
}
const mentionedPeople = matchPublicFigureMentions(text)
const gazetteerThemes = classifySpeechByGazetteer({ transcript: text }).topics
try {
return await withPayloadTransaction(payload, async ({ req }) => {
// Fresh read inside the transaction: whatever the assessoria curated
// (or another run catalogued) while this photo was in flight wins.
const fresh = await payload.find({
collection: ARCHIVE_PHOTO_SLUG,
where: { id: { equals: item.id } },
depth: 0,
limit: 1,
pagination: false,
select: { alt: true, curatedFields: true, catalog: true },
req,
// Intentional bypass: the catalogue CLI is a trusted actor with no session.
overrideAccess: true,
})
const doc = fresh.docs[0]
if (!doc) return { status: 'failed', stage: 'write', error: 'Foto não encontrada.' }
if (doc.catalog?.catalogedAt) return { status: 'skipped' }
const write = buildArchivePhotoCatalogWrite({
suggestion,
municipalityId,
mentionedPeople,
gazetteerThemes,
curatedFields: doc.curatedFields ?? [],
// Depth 0: the relationship is the id even when the generated type
// still allows the populated shape.
currentCatalog: doc.catalog
? {
...doc.catalog,
municipality:
typeof doc.catalog.municipality === 'number' ? doc.catalog.municipality : null,
}
: null,
currentAlt: doc.alt,
catalogedAt: new Date().toISOString(),
})
await payload.update({
collection: ARCHIVE_PHOTO_SLUG,
id: item.id,
data: write.data,
depth: 0,
req,
context: { archivePhotoCatalog: true },
// Intentional bypass: the catalogue CLI is a trusted actor with no session.
overrideAccess: true,
})
return { status: 'cataloged', source: write.source }
})
} catch (error) {
return { status: 'failed', stage: 'write', error: messageOf(error, 'erro desconhecido') }
}
} finally {
if (tempDir) await rm(tempDir, { recursive: true, force: true }).catch(() => undefined)
}
}

View file

@ -0,0 +1,38 @@
import 'server-only'
import type { Payload } from 'payload'
import { matchMunicipalityMentions } from '@/lib/speechGazetteer'
/**
* C232 — the shared, conservative município resolution from free text: the
* gazetteer mentions are reduced to distinct cities and only a city with
* exactly ONE catalogue entry resolves. Salvador has one entry per TSE zone,
* so a bare "Salvador" mention is ambiguous and stays empty — the ficha picks
* the zone. Extracted from the content-piece cataloguing (C211) so the photo
* cataloguing (C232) uses the same rule instead of a twin.
*/
export const resolveMentionedMunicipalityId = async ({
payload,
text,
}: {
payload: Payload
text: string
}): Promise<number | null> => {
const mentions = matchMunicipalityMentions(text)
const cities = new Set(mentions.map((entry) => entry.city))
if (cities.size !== 1) return null
const entries = mentions.filter((entry) => entry.city === [...cities][0])
if (entries.length !== 1) return null
const found = await payload.find({
collection: 'municipality',
where: { slug: { equals: entries[0]!.slug } },
depth: 0,
limit: 1,
pagination: false,
// Intentional admin bypass: the município catalog is read-only geography.
overrideAccess: true,
})
return found.docs[0]?.id ?? null
}

View file

@ -0,0 +1,337 @@
// @vitest-environment node
import { mkdtemp, rm, writeFile } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import type { Payload } from 'payload'
import { getPayload } from 'payload'
import { afterAll, beforeAll, describe, expect, it } from 'vitest'
import { ARCHIVE_PHOTO_SLUG, type ArchivePhotoImport } from '@/lib/archivePhoto'
import type { ArchivePhotoSuggestion } from '@/lib/archivePhotoCatalog'
import { publicFigureCatalog } from '@/lib/publicFigureCatalog'
import config from '@/payload.config'
import {
catalogArchivePhoto,
listArchivePhotoCatalogQueue,
type ArchivePhotoAnalyzer,
type ArchivePhotoCatalogItem,
} from '@/utilities/flickr/archivePhotoCatalog'
import { ingestArchivePhoto } from '@/utilities/flickr/archivePhotoIngest'
import { ARCHIVE_PHOTO_JPEG_BYTES } from '../helpers/archivePhotoFixture'
import { installCampaignFixtures } from '../helpers/campaignFixtures'
// C232 — the cataloguing boundary over the real Payload `teqo_test`: the
// transactional write, the curated-wins rule, the honest failure (nothing
// written, retried) and the derived index (searchText/takenOn). The vision
// engine is a stub — the real provider never enters the suite.
let payload: Payload
const createdFlickrIds = new Set<string>()
const tempDirs: string[] = []
const campaignFixtures = installCampaignFixtures({
getPayload: () => payload,
setPayload: (nextPayload) => {
payload = nextPayload
},
})
const figureName = publicFigureCatalog[0]!.name
const record = (
flickrId: string,
overrides: Partial<ArchivePhotoImport> = {},
): ArchivePhotoImport => {
createdFlickrIds.add(flickrId)
return {
flickrId,
sourceUrl: `https://www.flickr.com/photos/depjorgesolla/${flickrId}/`,
owner: '12345678@N00',
license: '0',
title: `Foto ${flickrId}`,
description: null,
tags: [],
takenAt: '2025-03-14 10:20:30',
postedAt: '2025-03-15T10:00:00.000Z',
albums: [],
geo: null,
exif: [],
originalUrl: `https://live.staticflickr.com/65535/${flickrId}_abcdef_o.jpg`,
originalKind: 'original',
...overrides,
}
}
const createPhoto = async (
flickrId: string,
overrides: Partial<ArchivePhotoImport> = {},
): Promise<{ id: number; item: ArchivePhotoCatalogItem }> => {
const dir = await mkdtemp(join(tmpdir(), 'archive-photo-'))
tempDirs.push(dir)
const filePath = join(dir, `flickr-${flickrId}.jpg`)
await writeFile(filePath, ARCHIVE_PHOTO_JPEG_BYTES)
const created = await ingestArchivePhoto(payload, record(flickrId, overrides), { filePath })
if (created.status !== 'created') throw new Error(`fixture not created: ${created.status}`)
const queue = await listArchivePhotoCatalogQueue({ payload })
const item = queue.find((entry) => entry.flickrId === flickrId)
if (!item) throw new Error(`fixture not in the catalog queue: ${flickrId}`)
return { id: created.id, item }
}
const suggestion = (overrides: Partial<ArchivePhotoSuggestion> = {}): ArchivePhotoSuggestion => ({
caption: null,
description: null,
scene: null,
visibleText: null,
hasPeople: null,
themes: [],
...overrides,
})
const analyzeWith =
(answer: ArchivePhotoSuggestion): ArchivePhotoAnalyzer =>
async () =>
answer
const findByFlickrId = async (flickrId: string) => {
const found = await payload.find({
collection: ARCHIVE_PHOTO_SLUG,
where: { flickrId: { equals: flickrId } },
depth: 0,
limit: 1,
overrideAccess: true,
})
return found.docs[0] ?? null
}
const municipalityId = async (slug: string): Promise<number> => {
const found = await payload.find({
collection: 'municipality',
where: { slug: { equals: slug } },
depth: 0,
limit: 1,
overrideAccess: true,
})
const id = found.docs[0]?.id
if (id == null) throw new Error(`município fixture ausente: ${slug}`)
return id
}
describe('catalogArchivePhoto (C232)', () => {
beforeAll(async () => {
payload = await getPayload({ config: await config })
})
afterAll(async () => {
for (const flickrId of createdFlickrIds) {
await payload.delete({
collection: ARCHIVE_PHOTO_SLUG,
where: { flickrId: { equals: flickrId } },
overrideAccess: true,
})
}
await Promise.all(tempDirs.map((dir) => rm(dir, { recursive: true, force: true })))
})
it('catalogues one photo and derives the searchable index', async () => {
const feira = await municipalityId('feira-de-santana')
const { item } = await createPhoto('53900000001', {
title: 'Plenária em Feira de Santana',
albums: [{ albumId: '1', title: 'Plenária em Feira de Santana' }],
tags: ['saude'],
})
const result = await catalogArchivePhoto({
payload,
item,
analyze: analyzeWith(
suggestion({
caption: 'Plenária lotada no interior',
description: 'Solla discursa para uma plateia cheia.',
scene: 'plenaria',
visibleText: 'SAÚDE É DIREITO',
hasPeople: true,
themes: ['saude'],
}),
),
})
expect(result).toEqual({ status: 'cataloged', source: 'ai' })
const doc = await findByFlickrId('53900000001')
expect(doc?.catalog).toMatchObject({
caption: 'Plenária lotada no interior',
description: 'Solla discursa para uma plateia cheia.',
scene: 'plenaria',
visibleText: 'SAÚDE É DIREITO',
hasPeople: true,
themes: ['saude'],
municipality: feira,
source: 'ai',
})
expect(doc?.catalog?.catalogedAt).toBeTruthy()
expect(doc?.takenOn).toBe('2025-03-14T12:00:00.000Z')
expect(doc?.searchText).toContain('plenaria')
expect(doc?.searchText).toContain('feira de santana')
expect(doc?.searchText).toContain('saude')
expect(doc?.searchText).toContain('plenaria lotada no interior')
})
it('matches the curated public-figure catalog against the text, never the model', async () => {
const { item } = await createPhoto('53900000002', {
title: `Encontro com ${figureName}`,
})
expect(item.title).toContain(figureName)
const result = await catalogArchivePhoto({
payload,
item,
analyze: analyzeWith(suggestion({ caption: 'Encontro político' })),
})
expect(result).toEqual({ status: 'cataloged', source: 'ai' })
const doc = await findByFlickrId('53900000002')
expect(doc?.catalog?.people).toEqual([figureName])
})
it('states metadata when the model has nothing to propose', async () => {
const feira = await municipalityId('feira-de-santana')
const { item } = await createPhoto('53900000003', {
title: 'Visita a Feira de Santana',
})
const result = await catalogArchivePhoto({
payload,
item,
analyze: analyzeWith(suggestion()),
})
expect(result).toEqual({ status: 'cataloged', source: 'metadata' })
const doc = await findByFlickrId('53900000003')
expect(doc?.catalog?.municipality).toBe(feira)
expect(doc?.catalog?.source).toBe('metadata')
expect(doc?.catalog?.catalogedAt).toBeTruthy()
})
it('states none, honestly, when there is nothing at all', async () => {
const { item } = await createPhoto('53900000004')
const result = await catalogArchivePhoto({
payload,
item,
analyze: analyzeWith(suggestion()),
})
expect(result).toEqual({ status: 'cataloged', source: 'none' })
const doc = await findByFlickrId('53900000004')
expect(doc?.catalog?.source).toBe('none')
expect(doc?.catalog?.catalogedAt).toBeTruthy()
})
it('never overwrites a curated field', async () => {
const feira = await municipalityId('feira-de-santana')
const { id, item } = await createPhoto('53900000005', {
title: 'Plenária em Feira de Santana',
})
const admin = await campaignFixtures().createAdminUser()
await payload.update({
collection: ARCHIVE_PHOTO_SLUG,
id,
data: { catalog: { caption: 'Legenda humana', municipality: feira } },
user: admin,
overrideAccess: false,
})
const result = await catalogArchivePhoto({
payload,
item,
analyze: analyzeWith(suggestion({ caption: 'Legenda da IA' })),
})
expect(result).toEqual({ status: 'cataloged', source: 'none' })
const doc = await findByFlickrId('53900000005')
expect(doc?.catalog?.caption).toBe('Legenda humana')
expect(doc?.catalog?.municipality).toBe(feira)
expect(doc?.curatedFields).toEqual(expect.arrayContaining(['caption', 'municipality']))
})
it('reports a failed analysis, writes nothing and lets the next run retry', async () => {
const { item } = await createPhoto('53900000006', { title: 'Plenária em Feira de Santana' })
const failing: ArchivePhotoAnalyzer = async () => {
throw new Error('engine fora do ar')
}
const failed = await catalogArchivePhoto({ payload, item, analyze: failing })
expect(failed).toEqual({ status: 'failed', stage: 'analyze', error: 'engine fora do ar' })
expect((await findByFlickrId('53900000006'))?.catalog?.catalogedAt ?? null).toBeNull()
const retried = await catalogArchivePhoto({
payload,
item,
analyze: analyzeWith(suggestion({ caption: 'Segunda tentativa' })),
})
expect(retried).toEqual({ status: 'cataloged', source: 'ai' })
expect((await findByFlickrId('53900000006'))?.catalog?.caption).toBe('Segunda tentativa')
})
it('skips a photo catalogued after the queue was read', async () => {
const { item } = await createPhoto('53900000007')
await catalogArchivePhoto({
payload,
item,
analyze: analyzeWith(suggestion({ caption: 'Primeira passada' })),
})
const second = await catalogArchivePhoto({
payload,
item,
analyze: analyzeWith(suggestion({ caption: 'Segunda passada' })),
})
expect(second).toEqual({ status: 'skipped' })
expect((await findByFlickrId('53900000007'))?.catalog?.caption).toBe('Primeira passada')
expect(
(await listArchivePhotoCatalogQueue({ payload })).map((entry) => entry.flickrId),
).not.toContain('53900000007')
})
it('caps the queue with the canary limit and marks curation only on user edits', async () => {
const { id } = await createPhoto('53900000008')
await createPhoto('53900000009')
const queued = await listArchivePhotoCatalogQueue({ payload, limit: 1 })
expect(queued).toHaveLength(1)
const admin = await campaignFixtures().createAdminUser()
await payload.update({
collection: ARCHIVE_PHOTO_SLUG,
id,
data: { catalog: { scene: 'evento' } },
user: admin,
overrideAccess: false,
})
expect((await findByFlickrId('53900000008'))?.curatedFields).toEqual(['scene'])
// A no-op admin save marks nothing new.
await payload.update({
collection: ARCHIVE_PHOTO_SLUG,
id,
data: { catalog: { scene: 'evento' } },
user: admin,
overrideAccess: false,
})
expect((await findByFlickrId('53900000008'))?.curatedFields).toEqual(['scene'])
// The system write (the cataloguing itself) never marks.
await payload.update({
collection: ARCHIVE_PHOTO_SLUG,
id,
data: { catalog: { caption: 'escrita do sistema' } },
context: { archivePhotoCatalog: true },
overrideAccess: true,
})
expect((await findByFlickrId('53900000008'))?.curatedFields).toEqual(['scene'])
})
})

View file

@ -0,0 +1,125 @@
import { spawnSync } from 'node:child_process'
import { join } from 'node:path'
import { describe, expect, it } from 'vitest'
// C232 — the cataloguing CLI's fail-closed write guards, exercised as a
// subprocess: each refusal must happen before any engine call or DB connection.
const repoRoot = process.cwd()
const scriptPath = join(repoRoot, 'scripts', 'catalog-archive-photos.mjs')
const run = (args: string[], env: Record<string, string> = {}) =>
spawnSync(process.execPath, [scriptPath, ...args], {
cwd: repoRoot,
encoding: 'utf8',
timeout: 30_000,
env: {
...process.env,
NODE_OPTIONS: '--no-deprecation --import=tsx/esm --import=./scripts/seed-loader.mjs',
NODE_ENV: 'production',
VITEST: '',
DATABASE_URL: 'postgresql://teqo:teqo@127.0.0.1:5432/teqo_1313',
PAYLOAD_SECRET: 'test-secret',
ARCHIVE_CATALOG_CONFIRM: '',
ARCHIVE_VISION_BASE_URL: '',
ARCHIVE_VISION_MODEL: '',
ARCHIVE_VISION_ALLOW_REMOTE: '',
TEQO_ENV: '',
ALLOW_REMOTE_DB: '',
S3_BUCKET: '',
S3_ENDPOINT: '',
S3_ACCESS_KEY_ID: '',
S3_SECRET_ACCESS_KEY: '',
...env,
},
})
const output = (result: ReturnType<typeof run>) => `${result.stdout}${result.stderr}`
const withS3 = {
S3_BUCKET: 'bucket',
S3_ENDPOINT: 'http://127.0.0.1:3900',
S3_ACCESS_KEY_ID: 'key',
S3_SECRET_ACCESS_KEY: 'secret',
}
describe('archive:catalog write guards (C232)', () => {
it('prints the help even without a database', () => {
const result = run(['--help'], { DATABASE_URL: '' })
expect(result.status).toBe(0)
expect(result.stdout).toContain('ARCHIVE_CATALOG_CONFIRM')
expect(result.stdout).toContain('ARCHIVE_VISION_BASE_URL')
})
it('refuses --apply on a production target without the intent flag', () => {
const result = run(['--apply'])
expect(result.status).toBe(1)
expect(output(result)).toContain('ARCHIVE_CATALOG_CONFIRM=1')
})
it('requires the declared TEQO_ENV once the intent flag is set', () => {
const result = run(['--apply'], { ARCHIVE_CATALOG_CONFIRM: '1' })
expect(result.status).toBe(1)
expect(output(result)).toContain('TEQO_ENV')
})
it('refuses a target database that does not match the declared environment', () => {
const result = run(['--apply'], { ARCHIVE_CATALOG_CONFIRM: '1', TEQO_ENV: 'staging' })
expect(result.status).toBe(1)
expect(output(result)).toContain('≠ "teqo_staging"')
})
it('refuses --apply without the complete S3_* set on a declared production target', () => {
const result = run(['--apply'], { ARCHIVE_CATALOG_CONFIRM: '1', TEQO_ENV: 'production' })
expect(result.status).toBe(1)
expect(output(result)).toContain('S3_BUCKET')
})
it('refuses --apply without the vision endpoint', () => {
const result = run(['--apply'], {
ARCHIVE_CATALOG_CONFIRM: '1',
TEQO_ENV: 'production',
...withS3,
})
expect(result.status).toBe(1)
expect(output(result)).toContain('ARCHIVE_VISION_BASE_URL')
})
it('refuses a public vision host without the explicit remote escape', () => {
const result = run(['--apply'], {
ARCHIVE_CATALOG_CONFIRM: '1',
TEQO_ENV: 'production',
ARCHIVE_VISION_BASE_URL: 'https://api.example.com/v1',
...withS3,
})
expect(result.status).toBe(1)
expect(output(result)).toContain('host público')
expect(output(result)).toContain('ARCHIVE_VISION_ALLOW_REMOTE')
})
it('requires the model once the private endpoint is accepted', () => {
const result = run(['--apply'], {
ARCHIVE_CATALOG_CONFIRM: '1',
TEQO_ENV: 'production',
ARCHIVE_VISION_BASE_URL: 'http://100.94.122.26:11434/v1',
...withS3,
})
expect(result.status).toBe(1)
expect(output(result)).toContain('ARCHIVE_VISION_MODEL')
})
it('refuses an unknown argument before touching anything', () => {
const result = run(['--force'])
expect(result.status).toBe(1)
expect(output(result)).toContain('argumento desconhecido')
})
})

View file

@ -0,0 +1,141 @@
import { describe, expect, it } from 'vitest'
import {
ARCHIVE_CATALOG_DEFAULT_OUT_DIR,
formatArchiveCatalogInventory,
formatArchiveCatalogReport,
parseArchiveCatalogCliArgs,
summarizeArchiveCatalogInventory,
summarizeArchiveCatalogResults,
} from '../../scripts/lib/archiveCatalogPlan.mjs'
// C232 — the pure CLI planning/reporting of the archive cataloguing: the argv
// contract, the run summaries and the `--verify` inventory the receipt and the
// stdout lines are built from.
describe('parseArchiveCatalogCliArgs (C232)', () => {
it('defaults to the plan mode and the canonical receipt dir', () => {
expect(parseArchiveCatalogCliArgs([])).toEqual({
apply: false,
verify: false,
limit: null,
out: ARCHIVE_CATALOG_DEFAULT_OUT_DIR,
help: false,
})
})
it('parses the modes and options', () => {
expect(
parseArchiveCatalogCliArgs(['--apply', '--limit', '5', '--out', 'data/x']),
).toMatchObject({
apply: true,
limit: 5,
out: 'data/x',
})
expect(parseArchiveCatalogCliArgs(['--verify'])).toMatchObject({ verify: true })
expect(parseArchiveCatalogCliArgs(['--help'])).toMatchObject({ help: true })
})
it('refuses conflicting, unknown and unsafe input', () => {
expect(() => parseArchiveCatalogCliArgs(['--apply', '--verify'])).toThrow(/mutuamente/)
expect(() => parseArchiveCatalogCliArgs(['--limit', '0'])).toThrow(/--limit/)
expect(() => parseArchiveCatalogCliArgs(['--limit'])).toThrow(/faltou valor/)
expect(() => parseArchiveCatalogCliArgs(['--force'])).toThrow(/desconhecido/)
expect(() => parseArchiveCatalogCliArgs(['--out', '../fora'])).toThrow(/\.\./)
})
})
describe('summarizeArchiveCatalogResults (C232)', () => {
it('counts the outcomes, the sources and names every failure', () => {
const summary = summarizeArchiveCatalogResults([
{ flickrId: '1', status: 'cataloged', source: 'ai' },
{ flickrId: '2', status: 'cataloged', source: 'none' },
{ flickrId: '3', status: 'skipped' },
{ flickrId: '4', status: 'failed', stage: 'analyze', error: 'engine fora do ar' },
])
expect(summary).toEqual({
cataloged: 2,
skipped: 1,
failed: 1,
bySource: { ai: 1, metadata: 0, none: 1 },
failures: [{ flickrId: '4', stage: 'analyze', error: 'engine fora do ar' }],
})
})
})
describe('summarizeArchiveCatalogInventory (C232)', () => {
it('separates catalogued from pending and flags the missing file', () => {
const inventory = summarizeArchiveCatalogInventory([
{
flickrId: '1',
filename: 'a.jpg',
catalog: { catalogedAt: 'x', source: 'ai', scene: 'evento', municipality: 1 },
},
{ flickrId: '2', filename: 'b.jpg', catalog: { catalogedAt: 'x', source: 'metadata' } },
{ flickrId: '3', filename: 'c.jpg', catalog: {} },
{ flickrId: '4', filename: null, catalog: { catalogedAt: 'x', source: 'none' } },
])
expect(inventory).toMatchObject({
total: 4,
cataloged: 3,
pending: 1,
bySource: { ai: 1, metadata: 1, none: 1 },
withoutScene: 2,
withoutMunicipality: 2,
missingFilename: ['4'],
})
})
})
describe('report lines (C232)', () => {
it('renders the plan line with the pending count', () => {
const lines = formatArchiveCatalogReport({
mode: 'plan',
target: '127.0.0.1/teqo_staging',
engine: { host: '100.94.122.26', model: 'qwen2.5vl:7b', scope: 'local' },
pending: 12,
durationMs: 1500,
})
expect(lines[0]).toContain('100.94.122.26 (qwen2.5vl:7b, local)')
expect(lines.join('\n')).toContain('pendentes na fila: 12')
})
it('renders the apply summary and every failure', () => {
const lines = formatArchiveCatalogReport({
mode: 'apply',
target: '127.0.0.1/teqo_staging',
durationMs: 2000,
summary: summarizeArchiveCatalogResults([
{ flickrId: '1', status: 'cataloged', source: 'ai' },
{ flickrId: '2', status: 'failed', stage: 'analyze', error: 'timeout' },
]),
})
const text = lines.join('\n')
expect(text).toContain('processadas: 1')
expect(text).toContain('falharam: 1')
expect(text).toContain('IA: 1')
expect(text).toContain('2 (analyze): timeout')
})
it('renders the verify inventory with the pending gap', () => {
const lines = formatArchiveCatalogInventory({
total: 3,
cataloged: 2,
pending: 1,
bySource: { ai: 1, metadata: 1, none: 0 },
withoutScene: 1,
withoutMunicipality: 1,
missingFilename: [],
})
const text = lines.join('\n')
expect(text).toContain('catalogadas: 2')
expect(text).toContain('pendentes: 1')
expect(text).toContain('IA: 1')
expect(text).toContain('ainda pendentes de catalogação: 1')
})
})

View file

@ -0,0 +1,357 @@
import { describe, expect, it } from 'vitest'
import {
ARCHIVE_PHOTO_CAPTION_MAX_LENGTH,
archivePhotoSearchText,
archivePhotoTakenOn,
archivePhotoThemesFrom,
buildArchivePhotoCatalogWrite,
changedArchivePhotoCuratedFields,
matchPublicFigureMentions,
parseArchiveVisionOutput,
resolveArchivePhotoScene,
} from '@/lib/archivePhotoCatalog'
import { publicFigureCatalog } from '@/lib/publicFigureCatalog'
// C232 — the pure rules of the archive pre-cataloguing: closed vocabularies
// (scene/themes/curated fields), the fail-closed vision parser, the
// deterministic text matches and the merge that never overwrites curation.
// No provider, network or Payload here.
const suggestion = (
overrides: Partial<Parameters<typeof buildArchivePhotoCatalogWrite>[0]['suggestion']> = {},
) => ({
caption: null as string | null,
description: null as string | null,
scene: null as ReturnType<typeof resolveArchivePhotoScene>,
visibleText: null as string | null,
hasPeople: null as boolean | null,
themes: [] as ReturnType<typeof archivePhotoThemesFrom>,
...overrides,
})
const writeInput = (
overrides: Partial<Parameters<typeof buildArchivePhotoCatalogWrite>[0]> = {},
): Parameters<typeof buildArchivePhotoCatalogWrite>[0] => ({
suggestion: suggestion(),
municipalityId: null,
mentionedPeople: [],
gazetteerThemes: [],
curatedFields: [],
currentCatalog: null,
currentAlt: null,
catalogedAt: '2026-09-26T12:00:00.000Z',
...overrides,
})
describe('resolveArchivePhotoScene (C232)', () => {
it('resolves value and pt-BR label, accent/case insensitive', () => {
expect(resolveArchivePhotoScene('plenaria')).toBe('plenaria')
expect(resolveArchivePhotoScene('Plenária')).toBe('plenaria')
expect(resolveArchivePhotoScene('SEGURANCA')).toBeNull()
expect(resolveArchivePhotoScene('Audiência')).toBe('audiencia')
expect(resolveArchivePhotoScene('outro')).toBe('outro')
})
it('is fail-closed for unknown or non-string tokens', () => {
expect(resolveArchivePhotoScene('comício')).toBeNull()
expect(resolveArchivePhotoScene(null)).toBeNull()
expect(resolveArchivePhotoScene(7)).toBeNull()
expect(resolveArchivePhotoScene('')).toBeNull()
})
})
describe('archivePhotoThemesFrom (C232)', () => {
it('resolves the existing taxonomy and drops the unknown', () => {
expect(archivePhotoThemesFrom(['saude', 'Saúde', 'educacao', 'inventado', 7])).toEqual([
'saude',
'educacao',
])
expect(archivePhotoThemesFrom([])).toEqual([])
})
})
describe('matchPublicFigureMentions (C232)', () => {
const name = publicFigureCatalog[0]!.name
it('matches the curated catalog by whole word, accent/case insensitive', () => {
expect(matchPublicFigureMentions(`Plenária com ${name} no palco`)).toEqual([name])
expect(matchPublicFigureMentions(name.toLowerCase())).toEqual([name])
})
it('never invents a name and ignores unrelated text', () => {
expect(matchPublicFigureMentions('Plenária em Vitória da Conquista')).toEqual([])
expect(matchPublicFigureMentions('')).toEqual([])
})
})
describe('parseArchiveVisionOutput (C232)', () => {
it('parses a fenced JSON answer and normalizes the closed fields', () => {
const answer = [
'```json',
JSON.stringify({
caption: ' Plenária lotada no interior ',
description: 'Solla discursa para uma plateia cheia.',
scene: 'Plenária',
visibleText: 'SAÚDE É DIREITO',
hasPeople: true,
themes: ['Saúde', 'inventado'],
}),
'```',
].join('\n')
expect(parseArchiveVisionOutput(answer)).toEqual({
caption: 'Plenária lotada no interior',
description: 'Solla discursa para uma plateia cheia.',
scene: 'plenaria',
visibleText: 'SAÚDE É DIREITO',
hasPeople: true,
themes: ['saude'],
})
})
it('is null for a non-object answer (named failure upstream)', () => {
expect(parseArchiveVisionOutput('resposta em prosa')).toBeNull()
expect(parseArchiveVisionOutput('["a"]')).toBeNull()
expect(parseArchiveVisionOutput('{quebrado')).toBeNull()
expect(parseArchiveVisionOutput(null)).toBeNull()
})
it('degrades field by field instead of guessing', () => {
expect(
parseArchiveVisionOutput({
caption: 42,
scene: 'comício',
themes: ['saude', 5],
hasPeople: 'sim',
}),
).toEqual({
caption: null,
description: null,
scene: null,
visibleText: null,
hasPeople: null,
themes: ['saude'],
})
})
it('clips the caption to the field contract', () => {
const parsed = parseArchiveVisionOutput({ caption: 'a'.repeat(500) })
expect(parsed?.caption).toHaveLength(ARCHIVE_PHOTO_CAPTION_MAX_LENGTH)
})
})
describe('archivePhotoTakenOn (C232)', () => {
it('keeps the wall-clock day at noon UTC', () => {
expect(archivePhotoTakenOn('2025-03-14 10:20:30')).toBe('2025-03-14T12:00:00.000Z')
expect(archivePhotoTakenOn('2025-03-14')).toBe('2025-03-14T12:00:00.000Z')
})
it('is null for a missing or impossible date', () => {
expect(archivePhotoTakenOn(null)).toBeNull()
expect(archivePhotoTakenOn('14/03/2025')).toBeNull()
expect(archivePhotoTakenOn('2025-02-31 10:00:00')).toBeNull()
})
})
describe('changedArchivePhotoCuratedFields (C232)', () => {
it('marks only what actually changed for a request with data', () => {
expect(
changedArchivePhotoCuratedFields({
data: { alt: 'novo', catalog: { caption: 'x', scene: 'plenaria' } },
originalDoc: { alt: 'velho', catalog: { caption: 'x', scene: null } },
}),
).toEqual(['alt', 'scene'])
})
it('treats clearing and array edits as curation', () => {
expect(
changedArchivePhotoCuratedFields({
data: { catalog: { themes: [], people: ['Fulano'] } },
originalDoc: { catalog: { themes: ['saude'], people: ['Fulano'] } },
}),
).toEqual(['themes'])
})
it('normalizes the relationship shape and ignores absent fields', () => {
expect(
changedArchivePhotoCuratedFields({
data: { catalog: { municipality: { id: 5 } } },
originalDoc: { catalog: { municipality: 5 } },
}),
).toEqual([])
expect(
changedArchivePhotoCuratedFields({ data: {}, originalDoc: { catalog: { caption: 'x' } } }),
).toEqual([])
expect(changedArchivePhotoCuratedFields({ data: null, originalDoc: {} })).toEqual([])
})
})
describe('buildArchivePhotoCatalogWrite (C232)', () => {
it('writes the model fields and derives the alt from the caption', () => {
const { data, source } = buildArchivePhotoCatalogWrite(
writeInput({
suggestion: suggestion({
caption: 'Comício em Feira de Santana',
description: 'Solla cumprimenta apoiadores.',
scene: 'evento',
visibleText: 'JUNTOS PELA BAHIA',
hasPeople: true,
themes: ['politica-instituicoes'],
}),
}),
)
expect(source).toBe('ai')
expect(data.alt).toBe('Comício em Feira de Santana')
expect(data.catalog).toEqual({
caption: 'Comício em Feira de Santana',
description: 'Solla cumprimenta apoiadores.',
scene: 'evento',
visibleText: 'JUNTOS PELA BAHIA',
hasPeople: true,
themes: ['politica-instituicoes'],
source: 'ai',
catalogedAt: '2026-09-26T12:00:00.000Z',
})
})
it('states metadata when only the text-derived fields landed', () => {
const { data, source } = buildArchivePhotoCatalogWrite(
writeInput({
municipalityId: 7,
mentionedPeople: ['Fulano de Tal'],
gazetteerThemes: ['saude'],
}),
)
expect(source).toBe('metadata')
expect(data.alt).toBeUndefined()
expect(data.catalog).toEqual({
themes: ['saude'],
people: ['Fulano de Tal'],
municipality: 7,
source: 'metadata',
catalogedAt: '2026-09-26T12:00:00.000Z',
})
})
it('states none when there was nothing to propose', () => {
const { data, source } = buildArchivePhotoCatalogWrite(writeInput())
expect(source).toBe('none')
expect(data.catalog).toEqual({
source: 'none',
catalogedAt: '2026-09-26T12:00:00.000Z',
})
})
it('never overwrites a curated field, and the source reflects only what landed', () => {
const { data, source } = buildArchivePhotoCatalogWrite(
writeInput({
suggestion: suggestion({ caption: 'IA', themes: ['saude'] }),
municipalityId: 9,
curatedFields: ['caption', 'themes', 'municipality'],
currentCatalog: { caption: 'Humano', themes: ['esporte'], municipality: 3 },
}),
)
expect(source).toBe('none')
expect(data.catalog).toEqual({
caption: 'Humano',
themes: ['esporte'],
municipality: 3,
source: 'none',
catalogedAt: '2026-09-26T12:00:00.000Z',
})
expect(data.alt).toBeUndefined()
})
it('keeps writing when the curation freezes only one field', () => {
const { data, source } = buildArchivePhotoCatalogWrite(
writeInput({
suggestion: suggestion({ caption: 'IA', description: 'Descrição da IA' }),
curatedFields: ['caption'],
currentCatalog: { caption: 'Humano' },
currentAlt: 'Alt humano',
}),
)
expect(source).toBe('ai')
expect(data.catalog.caption).toBe('Humano')
expect(data.catalog.description).toBe('Descrição da IA')
// The curated caption freezes the alt refinement too (never mirrors human text).
expect(data.alt).toBeUndefined()
})
it('keeps a curated alt and never mirrors the curated caption into it', () => {
const { data } = buildArchivePhotoCatalogWrite(
writeInput({
suggestion: suggestion({ caption: 'IA' }),
curatedFields: ['alt', 'caption'],
currentCatalog: { caption: 'Humano' },
currentAlt: 'Alt humano',
}),
)
expect(data.alt).toBe('Alt humano')
expect(data.catalog.caption).toBe('Humano')
})
it('writes a negative hasPeople as a real proposal, never a blank string', () => {
const negative = buildArchivePhotoCatalogWrite(
writeInput({ suggestion: suggestion({ hasPeople: false }) }),
)
expect(negative.source).toBe('ai')
expect(negative.data.catalog.hasPeople).toBe(false)
const blank = buildArchivePhotoCatalogWrite(
writeInput({ suggestion: suggestion({ caption: ' ' }) }),
)
expect(blank.source).toBe('none')
expect(blank.data.catalog.caption).toBeUndefined()
})
it('unions the model themes with the gazetteer without duplicating', () => {
const { data, source } = buildArchivePhotoCatalogWrite(
writeInput({
suggestion: suggestion({ themes: ['saude'] }),
gazetteerThemes: ['saude', 'educacao'],
}),
)
expect(source).toBe('ai')
expect(data.catalog.themes).toEqual(['saude', 'educacao'])
})
})
describe('archivePhotoSearchText (C232)', () => {
it('normalizes every facet into one haystack', () => {
const text = archivePhotoSearchText({
alt: 'Comício',
title: 'Plenária',
description: 'Evento',
catalogDescription: 'Descrição IA',
caption: 'Legenda',
scene: 'plenaria',
visibleText: 'SAÚDE',
themes: ['saude'],
people: ['Fulano de Tal'],
municipalityName: 'Feira de Santana',
albumTitles: ['Vitória da Conquista'],
tagNames: ['Solla'],
})
expect(text).toContain('comicio')
expect(text).toContain('plenaria')
expect(text).toContain('saude')
expect(text).toContain('fulano de tal')
expect(text).toContain('feira de santana')
expect(text).toContain('vitoria da conquista')
expect(text).toContain('solla')
})
it('is empty for an empty input', () => {
expect(archivePhotoSearchText({})).toBe('')
})
})

View file

@ -0,0 +1,149 @@
// @vitest-environment node
import { describe, expect, it, vi } from 'vitest'
import {
ArchiveVisionError,
analyzeArchivePhotoVision,
} from '../../scripts/lib/archiveVisionApi.mjs'
// C232 — the vision client contract: request shape, auth header, transient
// retry, terminal 4xx, timeout/abort and the named refusals. `fetch` is always
// a fake — the real engine never enters the suite.
const jsonResponse = (body: unknown, status = 200) =>
({
ok: status >= 200 && status < 300,
status,
json: async () => body,
text: async () => JSON.stringify(body),
}) as Response
const fakeFetch = (impl: (url: string, init?: RequestInit) => Promise<Response>) =>
vi.fn(impl) as unknown as typeof fetch
const call = (fetchImpl: typeof fetch, overrides: Record<string, unknown> = {}) =>
analyzeArchivePhotoVision({
baseUrl: 'http://100.94.122.26:11434/v1',
model: 'qwen2.5vl:7b',
imageDataUrl: 'data:image/jpeg;base64,AAAA',
prompt: 'descreva a foto',
systemPrompt: 'responda em JSON',
fetchImpl,
retryAttempts: 1,
sleep: async () => undefined,
...overrides,
})
describe('analyzeArchivePhotoVision (C232)', () => {
it('refuses to call without the engine config or a prepared image', async () => {
await expect(
call(
fakeFetch(async () => jsonResponse({})),
{ baseUrl: ' ' },
),
).rejects.toThrow(/ARCHIVE_VISION_BASE_URL/)
await expect(
call(
fakeFetch(async () => jsonResponse({})),
{ model: '' },
),
).rejects.toThrow(/ARCHIVE_VISION_MODEL/)
await expect(
call(
fakeFetch(async () => jsonResponse({})),
{ imageDataUrl: 'http://x/foto.jpg' },
),
).rejects.toThrow(/data: URL/)
})
it('posts the OpenAI-compatible shape with the image and the key header', async () => {
const fetchImpl = fakeFetch(async () =>
jsonResponse({ choices: [{ message: { content: '{"caption":"ok"}' } }] }),
)
const content = await call(fetchImpl, { apiKey: 'segredo' })
expect(content).toBe('{"caption":"ok"}')
const [url, init] = (fetchImpl as unknown as { mock: { calls: [string, RequestInit][] } }).mock
.calls[0]
expect(url).toBe('http://100.94.122.26:11434/v1/chat/completions')
expect(init.method).toBe('POST')
const headers = init.headers as Record<string, string>
expect(headers.Authorization).toBe('Bearer segredo')
const body = JSON.parse(String(init.body))
expect(body.model).toBe('qwen2.5vl:7b')
expect(body.stream).toBe(false)
expect(body.messages[0]).toMatchObject({ role: 'system', content: 'responda em JSON' })
expect(body.messages[1].content[0]).toMatchObject({ type: 'text', text: 'descreva a foto' })
expect(body.messages[1].content[1]).toMatchObject({
type: 'image_url',
image_url: { url: 'data:image/jpeg;base64,AAAA' },
})
})
it('retries a transient 5xx with backoff and succeeds', async () => {
const sleeps: number[] = []
let attempts = 0
const fetchImpl = fakeFetch(async () => {
attempts += 1
if (attempts === 1) return jsonResponse({ error: 'sobrecarregado' }, 503)
return jsonResponse({ choices: [{ message: { content: 'ok' } }] })
})
const content = await call(fetchImpl, {
retryAttempts: 3,
retryBaseDelayMs: 100,
sleep: async (ms: number) => {
sleeps.push(ms)
},
})
expect(content).toBe('ok')
expect(attempts).toBe(2)
expect(sleeps).toEqual([100])
})
it('does not retry a terminal 4xx refusal and keeps the provider detail', async () => {
const fetchImpl = fakeFetch(async () => jsonResponse({ error: 'modelo desconhecido' }, 404))
await expect(call(fetchImpl, { retryAttempts: 3 })).rejects.toMatchObject({
name: 'ArchiveVisionError',
status: 404,
message: expect.stringContaining('modelo desconhecido'),
})
expect((fetchImpl as unknown as { mock: { calls: unknown[] } }).mock.calls).toHaveLength(1)
})
it('retries a network failure as transient', async () => {
let attempts = 0
const fetchImpl = fakeFetch(async () => {
attempts += 1
if (attempts === 1) throw new TypeError('fetch failed')
return jsonResponse({ choices: [{ message: { content: 'ok' } }] })
})
await expect(call(fetchImpl, { retryAttempts: 2 })).resolves.toBe('ok')
expect(attempts).toBe(2)
})
it('names an off-contract answer instead of guessing', async () => {
await expect(call(fakeFetch(async () => jsonResponse({ choices: [] })))).rejects.toBeInstanceOf(
ArchiveVisionError,
)
await expect(
call(
fakeFetch(
async () =>
({
ok: true,
status: 200,
json: async () => {
throw new Error('html')
},
}) as unknown as Response,
),
),
).rejects.toThrow(/não é JSON/)
})
})

View file

@ -0,0 +1,55 @@
import { describe, expect, it } from 'vitest'
import { isPrivateVisionHost } from '../../scripts/lib/cli.mjs'
// C232 — the vision-endpoint locality predicate behind the PII guardrail: only
// the private network (loopback, RFC1918, CGNAT/tailnet, link-local/ULA, local
// names) passes without the explicit remote escape.
describe('isPrivateVisionHost (C232)', () => {
it('accepts loopback, private ranges and the tailnet CGNAT block', () => {
for (const host of [
'localhost',
'127.0.0.1',
'10.0.0.7',
'172.16.0.1',
'172.31.255.254',
'192.168.15.142',
'100.94.122.26',
'100.64.0.1',
'::1',
'[::1]',
'fe80::1',
'fd00::1',
'gpu-box',
'ollama',
'host.docker.internal',
'workstation.local',
'engine.internal',
]) {
expect(isPrivateVisionHost(host), host).toBe(true)
}
})
it('rejects public hosts and out-of-range addresses', () => {
for (const host of [
'8.8.8.8',
'1.1.1.1',
'172.32.0.1',
'172.15.0.1',
'100.63.0.1',
'100.128.0.1',
'203.0.113.10',
'api.openai.com',
'example.com',
'256.1.1.1',
// Names that merely START with the IPv6 ULA/link-local prefixes.
'fdic.gov',
'fcc.gov',
'fd-openai.com',
'',
]) {
expect(isPrivateVisionHost(host), host).toBe(false)
}
})
})