feat(C153): catálogo de falas do Solla — schema, access, import idempotente e busca

Collections speech/speechSegment no grupo Comunicação, papel communicator
(leitura restrita) e índice trigram normalizado para a busca por segmento.
Import idempotente da Câmara (discurso→evento→trecho→VOD→ASR) com facetas
gazetteer+LLM, proveniência e preservação de curadoria manual; parser de
quem presidia; reusa a lógica pura do piloto C152 via scripts/lib.

Related #955
This commit is contained in:
Francisco Solla 2026-09-12 21:39:11 -03:00
parent fa6933c1cd
commit 6c0261fd0f
30 changed files with 18709 additions and 111 deletions

View file

@ -0,0 +1,190 @@
# Impl: Catálogo de falas do Solla: import idempotente e busca textual
Status: em execução
Atualizado em: 2026-09-12
Issue: #955
Intenção: docs/plans/catalogo-falas-solla.md
Appetite restante: herdado (~2–3 dias; a Fase 4 é wall-clock de máquina, sem trabalho humano)
## Leitura da intenção
- **Outcome:** o acervo da 57ª legislatura existe no banco local com transcrição oficial, segmentos ASR com minutagem, facetas (tema/alcance) com proveniência e menções; o import é reexecutável sem duplicar e imprime relatório por execução; a busca por palavra sobre os segmentos acha com e sem acento. C155 leva a produção.
- **O que NÃO negociar:**
- Facetas com proveniência `gazetteer | llm | manual`; keyword oficial crua preservada (nunca substituída pela faceta).
- Leitura restrita a `communicator`, `coordinator` e `candidate` (+ Payload admin). `advisor` e `leader` não leem — nem via `/api`.
- Sem UI (C154), sem espelho de mídia/S3, sem normalizar pessoas citadas como `Contact`.
- Identidade do registro não é a URL do MP4 (regerável, hash não é contrato); import escreve só no DB local (C155 faz produção).
- **O que reavaliar (hipóteses da intenção):**
- "LLM do Sollinha via `DEEPSEEK_API_KEY`" — essa chave não existe no ambiente do import; o mesmo modelo (`deepseek-ai/DeepSeek-V4-Flash`) está verificado ao vivo na Deep Infra, provedor/chave que a ASR já usa. Reavaliado na D3.
- "chave natural por áudio+trecho" (C152) sozinha não cobre discurso sem trecho (54ª/sessões sem âncora). Ela vira **identidade de vídeo** (decide pular VOD/ASR), não chave do registro. Ver D2.
- "Direção no codebase: script no padrão `recover-media.mjs`" — o padrão mais próximo é `seed-tse-results.mjs` (importa `src/` via tsx + `seed-loader`, guard de banco local, `assertLocalDatabase`). É o que este plano usa.
## Abordagem recomendada
```mermaid
flowchart LR
subgraph Fonte Câmara
A[API discursos<br/>dadosabertos] --> B[selectSpeechEvents]
B --> C[event page cache<br/>data/camara/events]
C --> D[matchExcerpt]
C --> P[parsePresidingTransitions]
end
D --> E{VOD PRONTO?}
E -- sim --> F[MP4 cache<br/>data/camara]
F --> G[Deep Infra Whisper<br/>verbose_json segments]
E -- INDISPONIVEL / sem trecho --> R[registra sem segmentos]
A --> H[classifySpeech<br/>gazetteer + LLM]
G --> I[upsertSpeechBundle<br/>transação por sourceKey]
H --> I
P --> I
R --> I
I --> J[(speech + speechSegment)]
J --> K[searchText normalizado<br/>+ GIN pg_trgm]
```
**Opções consideradas:** A | B | C por decisão (D1–D6 abaixo).
**Recomendação:** as seis decisões, na ordem em que bloqueiam as demais; D1 e D2 travam o schema, D3–D5 travam o import e o access, D6 é dado de saída.
**Rejeitadas:** registradas dentro de cada decisão (formato obrigatório da decision-quality).
### Decisões de engenharia
#### D1 — Segmentos em collection própria, não array do discurso
- **Opções:** A) collection `speechSegment` (FK `speech`, `order`, `startSeconds`, `endSeconds`, `text`, `searchText`) | B) array de segmentos dentro de `speech` | C) um blob JSON em `speech`.
- **Recomendação:** A — a busca global (`contains` + GIN trgm) roda sobre uma tabela própria, paginável e `select`-enxuta; o C154 busca segmento e abre o discurso sem carregar 40–100 segmentos por doc. `speechSegment` fica admin-hidden (precedente `SupporterImportBatch.ts:26`) porque a superfície é o C154.
- **Rejeitadas:** B porque a query textual sobre array relacional do Payload não expõe `contains` por subcampo com o shape que o C154 precisa e infla todo read de discurso; C porque mata filtro por segmento e a indexação trigram.
#### D2 — `sourceKey` da identidade da API + identidade de vídeo armazenada
- **Opções:** A) `sourceKey = dataHoraInicio|tipoDiscurso|fase` (unique) e `eventId+audioId+excerptTMs` armazenados só para decidir re-transcrição | B) unique em `eventId+audioId+excerptTMs` | C) URL do MP4 como identidade.
- **Recomendação:** A — cobre discurso sem trecho (senão ele nunca teria chave) e sobrevive à regeneração do VOD; o import acha por `sourceKey`, e se `audioId+excerptTMs` baterem com o que já está no banco **e houver segmentos**, pula VOD+ASR e só atualiza metadados. `--skip-transcribe`/ASR falho passam `segments: undefined` (preserva os antigos, não zera).
- **Rejeitadas:** B porque discurso sem trecho ficaria de fora ou duplicaria a cada execução; C porque o `<hash>.mp4` é regerável e a permanência só foi medida em janela curta (C152).
#### D3 — Classificador LLM via Deep Infra raw fetch (mesmo provedor da ASR)
- **Opções:** A) raw fetch OpenAI-compat na Deep Infra, `model: deepseek-ai/DeepSeek-V4-Flash`, `response_format: json_object`, zod + fallback gazetteer | B) `@ai-sdk/deepseek` contra `api.deepseek.com` | C) só gazetteer.
- **Recomendação:** A — entrega a opção A da intenção (DeepSeek flash + validação + fallback C) sem segundo provedor nem segredo novo: a `DEEPINFRA_API_KEY` já existe no env e é a mesma da transcrição; custo medido US$ 0,00007/discurso (234 ≈ US$ 0,016). Saída validada por zod; HTTP erro/timeout/parse inválido → `classifiedBy='gazetteer'` com topics/municípios do lexicon (scopes vazios se o lexicon não achar termo). `classifiedBy='manual'` nunca é sobrescrito (D2/hook abaixo).
- **Rejeitadas:** B porque exige `DEEPSEEK_API_KEY` (ausente), acopla um SDK ao CLI para uma chamada única e cria dois caminhos de LLM no repo; C porque perde `scopes` (Bahia/Brasil/Internacional) e pessoas/programas, que são aceite de produto.
#### D4 — Busca: `searchText` normalizado + GIN `pg_trgm`
- **Opções:** A) coluna `searchText` normalizada app-side (NFD sem acento + lowercase) + índice GIN `gin_trgm_ops` hand-written | B) extensão `unaccent` + `ILIKE unaccent(text)` | C) `tsvector` com dicionário português.
- **Recomendação:** A — `contains` funciona igual em dev/test/prod sem extensão nova além do `pg_trgm` já instalado (precedente `20260719_020000_add_contact_trgm_index.ts`); a normalização é pura, unit-testada em `src/lib/speechSearch.ts`, e o C154 importa a MESMA função para montar a query. Hook `beforeValidate` em `speechSegment` deriva `searchText` de `text` (criação e edição manual).
- **Rejeitadas:** B porque `unaccent(text)` não é IMMUTABLE para índice funcional sem wrapper, depende de extension adicional e collation vira pegadinha; C porque `tsvector` faz stemming (piora prefixo/substring), exige config de dicionário e a semântica pedida é substring com/sem acento, não ranking lexical.
#### D5 — Papel `communicator` com predicado próprio, fora de `isStaffCampaignRole`
- **Opções:** A) novo valor no enum `campaignUser.role` + `canReadSpeechCatalog(role)` em `src/lib/campaignRoles.ts` + módulo `src/utilities/access/speeches.ts` reexportado por `campaignAccess.ts` | B) reusar `advisor`/staff | C) criar o leitor no `users` (admin Payload).
- **Recomendação:** A — o assessor de comunicação lê o catálogo sem herdar nenhuma área de staff: `isStaffCampaignRole` (`src/lib/campaignRoles.ts:13`) fica intocado. Leitura: Payload admin **ou** campaignUser `communicator|coordinator|candidate`. Update (correção manual de facetas): `coordinator|candidate|admin`. Create/delete: `payloadAdminOnly`. Import CLI escreve com `overrideAccess: true` (comentário "bypass", padrão seed-tse). Guardas mínimas de shell para o papel não vazar staff: entrada em `campaignRoleLabels` (`campaignUserProfile.ts:9` — o `Record` quebra o typecheck sem ela), branch explícito em `getCampaignNav` (`nav.ts:85`, hoje devolveria `staffNav` a qualquer não-leader; `communicator` → nav vazia até o C154), e nenhuma mudança em `homeActionsForRole` (já devolve `[]` para não-staff, verificado em `campaignHomeActions.ts:124`).
- **Rejeitadas:** B porque abriria o /campanha inteiro (viola "leitura restrita"); C porque o perfil de comunicação não é editor do CMS e ganharia o `/admin` todo.
#### D6 — "Quem presidia": heurística "Troca da mesa" na página do evento
- **Opções:** A) parse dos headings `g-l-assista__categoria-outros-videos` "Troca da mesa Presidente X por Participante Y" (transição = maior `t` do grupo; presidente em T = Y da transição mais recente ≤ T; antes da 1ª = X da mais antiga; sem transições = null) | B) deixar `presidingOfficer` sempre null | C) tirar do Diário (`urlTexto`) ou de outro endpoint.
- **Recomendação:** A — é a única fonte observada com o dado (API não tem campo); o parser entra no `scripts/lib/camaraSpeeches.mjs` (dono da leitura da página, reusado pelo import) como `parsePresidingOfficerTransitions`/`resolvePresidingOfficer`, com fixture de HTML e unit tests; layout mudou → parse devolve `[]`/null e o import nunca quebra.
- **Rejeitadas:** B porque joga fora dado pedido no aceite e já disponível; C porque `urlTexto` é a taquigrafia (não registra a mesa) e não há endpoint com o campo.
### Componentes / mudanças
- **`src/collections/Speech.ts`** (novo): slug `speech`, labels "Discurso"/"Discursos", `admin.group: 'Comunicação'`, `useAsTitle: 'speechAt'`, `admin.description` com o crédito CC BY 4.0 da Câmara. Campos: `sourceKey` (text, unique, index), `speechAt` (date, index), `year` (number, index; derivado por hook), `legislature` (select 54|55|56|57, index), `type`, `phase`, `durationSeconds` (number), `summary` (textarea), `officialTranscript` (textarea), `officialTextUrl`, `keywords` (text hasMany, cru), `eventId` (number, index), `eventType`, `eventStartAt`/`eventEndAt` (date), `youtubeUrl`, `audioId` (number, index), `excerptTMs` (number, index), `vodPlaybackUrl`/`vodDownloadUrl` (último conhecido, não identidade), `presidingOfficer`, `topics` (select hasMany, index, 18 valores de `SPEECH_TOPICS`), `scopes` (select hasMany, index, 3), `classifiedBy` (select `gazetteer|llm|manual`, default `gazetteer`), `mentionedMunicipalities` (relationship `municipality` hasMany), `mentionedPeople`/`mentionedPrograms`/`mentionedProjects` (text hasMany). Hooks: `beforeValidate` deriva `year`; `beforeChange` preserva facetas quando `originalDoc.classifiedBy === 'manual'` e a escrita é sem `req.user` (import/CLI) — edição autenticada no admin continua podendo corrigir e manter `manual`; `beforeDelete` apaga os segmentos do discurso (relacionamento Payload não tem FK). Access: `create/delete: payloadAdminOnly`, `read: canReadSpeech`, `update: canUpdateSpeech`.
- **`src/collections/SpeechSegment.ts`** (novo): slug `speechSegment`, labels "Segmento de fala", `admin.hidden: () => true` (precedente `SupporterImportBatch.ts:26`). Campos: `speech` (relationship, required, index), `order` (number, required), `startSeconds`/`endSeconds` (number, required), `text` (textarea, required), `searchText` (text, required, `admin.readOnly`, sem `index` — o GIN é hand-written). Hook `beforeValidate`: `searchText = normalizeForSearch(data.text ?? originalDoc?.text)`. Access espelha `speech` (read `canReadSpeech`; mutações `payloadAdminOnly`).
- **`src/payload.config.ts`**: registrar `Speech` e `SpeechSegment` no array `collections` (imports + entradas; grupo novo "Comunicação" aparece no admin).
- **`src/lib/speechFacets.ts`** (novo, puro/client-safe): contrato `SPEECH_TOPICS` (18 slugs pt — valores de dado, precedente `post.type`; nomes de código em inglês), `SPEECH_SCOPES`, `SPEECH_CLASSIFICATION_SOURCES`; lexicon gazetteer tema→termos; `matchTopicFacets(text)` com boundary de token (teste: "sus" NÃO casa "suspeito"); `matchMunicipalityMentions(text)` contra `municipalityCatalog` com consumo guloso do nome mais longo (testes: Conde vs Condeúba; São Félix vs São Félix do Coribe) — **menção a "Salvador" mapeia para as 19 zonas do catálogo** (a cidade não tem linha própria; a menção à cidade vale para qualquer zona, e o C154 filtra por município; documentar no campo/admin description). `parseLlmFacetResponse(json)` com zod (filtra para valores conhecidos, dedupe normalizado, caps de pessoas/programas/projetos); `mergeFacetClassification(gazetteer, llm)` (topics = união; scopes = do LLM validado ou do lexicon; `classifiedBy = llm ? 'llm' : 'gazetteer'`).
- **`src/lib/speechSearch.ts`** (novo, puro/client-safe): `normalizeForSearch(value)` (NFD, remove combining marks, lowercase, colapsa espaços) e `uniqueByNormalizedForm(values)`; importado pelo hook do segmento, pelo gazetteer e pelo C154.
- **`src/utilities/speech/speechImport.ts`** (novo, `server-only`): `findSpeechImportState(payload, sourceKey)` (leitura para o skip de VOD/ASR) e `upsertSpeechBundle(payload, bundle)` — reusa `withPayloadTransaction` (`src/utilities/payloadTransaction.ts:86`), acha por `sourceKey`, cria/atualiza, pula campos de faceta se existente é `manual`, substitui segmentos **somente quando `bundle.segments !== undefined`**, tudo com `req: { transactionID }` e `overrideAccess: true` com comentário "bypass". Retorna counts (created/updated/segmentsInserted/segmentsDeleted/manualPreserved) para o relatório.
- **`src/utilities/speech/speechClassifier.ts`** (novo, `server-only`): `classifySpeech(input)` com `DEEPINFRA_API_KEY`; raw fetch OpenAI-compat (`https://api.deepinfra.com/v1/openai/chat/completions`, `deepseek-ai/DeepSeek-V4-Flash`, `response_format: json_object`, `temperature: 0`, timeout), transcript truncado (≈12k chars) + keywords cruas; valida com `parseLlmFacetResponse`; qualquer falha/chave ausente → gazetteer-only. Nunca lança.
- **`src/utilities/access/speeches.ts`** (novo): `canReadSpeech` (admin ∪ campaignUser `communicator|coordinator|candidate`), `canUpdateSpeech` (admin ∪ coordinator/candidate); reexport no barrel `src/utilities/campaignAccess.ts` (prefixo `src/utilities/access` já mapeado no E2E manifest — sem mudança de manifesto).
- **`src/lib/campaignRoles.ts`**: `canReadSpeechCatalog(role)` client-safe (o predicado próprio, fora de `isStaffCampaignRole`).
- **`src/collections/CampaignUser.ts`**: opção `{ label: 'Assessor de Comunicação', value: 'communicator' }` no select de role (linhas 483–488).
- **`src/utilities/campaignUserProfile.ts`**: `communicator: 'Assessor de Comunicação'` em `campaignRoleLabels` (obrigatório para o typecheck pós-`generate:types`).
- **`src/components/campaign/shell/nav.ts`**: branch `if (role === 'communicator') return []` em `getCampaignNav` (nav vazia; C154 adiciona "Falas"). `getCampaignSecondaryNav`/`getCampaignBottomNav` já devolvem `[]` via `isStaffCampaignRole`.
- **`scripts/lib/camaraSpeeches.mjs`** (editar o dono): adicionar `parsePresidingOfficerTransitions(html)` e `resolvePresidingOfficer(transitions, tMs)` + export; reusar `parseEventExcerpts`, `buildVodUrl`, `parseVodStatus`, `selectSpeechEvents`, `matchExcerpt`, `normalizeTranscription`, `LEGISLATURE_RANGES`, `SOLLA_DEPUTY_ID`, `CAMARA_USER_AGENT`.
- **`scripts/import-camara-speeches.mjs`** (novo): `dieWithLabel('camara:import')`, `loadCliEnv`, `ensureCachedDownload`/`writeRepoFile` de `scripts/lib/cli.mjs`, `assertLocalDatabase`; flags `--legislature` (default 57) `--date` `--limit` `--skip-transcribe` `--reclassify` `--out` `--help`; cache de lista de eventos por dia e de página de evento em `data/camara/events/`, MP4 em `data/camara/` (já gitignored); prioriza `Sessão Deliberativa`/`Breves Comunicações` e capa a varredura de eventos por discurso (C152: ~50s/dia quando não há trecho); retry com backoff e delay educado; importa `speechImport`/`speechClassifier`/config via dynamic import literal (knip resolve); relatório no stdout + JSON em `data/camara/reports/<run>.json`; falha por discurso entra em `failures[]` e o run continua (resumível); exit 1 só em fatal (DB/guard).
- **`package.json`**: `"camara:import": "cross-env NODE_OPTIONS=\"--no-deprecation --import=tsx/esm --import=./scripts/seed-loader.mjs\" node scripts/import-camara-speeches.mjs"` (padrão `db:seed:tse`; `seed-loader` cobre `server-only` e `next/cache`).
- **Migrations:** `pnpm migrate:create add_speech_catalog` (rebase antes, conforme engineering-standards) → se o gerador não emitir, adicionar à mão `ALTER TYPE "public"."enum_campaign_user_role" ADD VALUE IF NOT EXISTS 'communicator'` (precedente `20260723_200000_remodel_municipalities.ts:105`; PG 12+ aceita dentro da transação e nada usa o valor nela; `down` irreversível documentado). Depois, hand-written `src/migrations/<ts>_add_speech_segment_trgm_index.ts` (`CREATE EXTENSION IF NOT EXISTS pg_trgm; CREATE INDEX IF NOT EXISTS "speech_segment_search_text_trgm_idx" ON "speech_segment" USING gin ("search_text" gin_trgm_ops);`, `down` dropa só o índice) registrado em `src/migrations/index.ts` como o `20260719_020000_add_contact_trgm_index`.
- **`pnpm generate:types`**: `payload-types.ts` ganha `Speech`/`SpeechSegment` e `'communicator'` no union de role.
- **Testes:** `tests/unit/speechFacets.unit.spec.ts`, `tests/unit/speechSearch.unit.spec.ts`, extensão de `tests/unit/camaraSpeeches.unit.spec.ts` (presiding + HTML fixture), `tests/int/speechCatalog.int.spec.ts` (access allow/deny + hook `searchText` + `contains`), `tests/int/speechImport.int.spec.ts` (idempotência, replace de segmentos, manual preservado, `segments: undefined` preserva, estado de skip).
- **Changelog:** `docs/changelog/2026-09-12-c153.md`.
### Dados → forma (se aplicável)
- **Sem UI** (N/A por decisão de produto). A "forma" deste item é o **relatório de execução** (stdout + JSON): `runAt`, `range`, `totals { listed, processed, created, updated, skippedExisting, withVideo, withSegments, withoutExcerpt, failed }`, `asr { calls, audioSeconds, estimatedCostUsd }` (US$ 0,00045/min), `llm { calls, totalTokens, estimatedCostUsd }`, `elapsedMs`, `failures[] { sourceKey, speechAt, stage: 'event'|'excerpt'|'vod'|'asr'|'classify'|'upsert', message }`.
- **Rejeitadas:** dashboard/admin como superfície (é o C154; aqui só dados), CSV de saída (o JSON do relatório já é legível por máquina), relatório por discurso no banco (ruído; o relatório é artefato de execução, não entidade).
## Fases verificáveis
1. **Tracer / schema+access+migration** (≈0,5 dia): collections registradas, papel `communicator` (option + label + predicado + nav), `access/speeches.ts` + reexport, migrations (catálogo + enum + trgm), `generate:types`. **Prova vertical (tracer cedo):** `tests/int/speechCatalog.int.spec.ts` cria 1 `speech` + 1 `speechSegment` via Local API, lê por `searchText contains 'saude'` com texto acentuado, e prova allow/deny por papel (communicator/coordinator/candidate/admin leem; advisor/leader/anônimo não; mutação só admin/coordinator/candidate). Aplicar migrations no DB de dev e no `_test` (skill `local-database`) antes do int.
2. **Módulos puros + unit** (≈0,5 dia): `speechFacets.ts`, `speechSearch.ts`, `canReadSpeechCatalog`; extensão do `camaraSpeeches.mjs` (presiding). Unit verde: boundary de tema, Conde/Condeúba, Salvador → 19 zonas, merge gazetteer+llm, zod rejeitando saída fora da taxonomia, normalização de busca, heurística da mesa com HTML congelado.
3. **Import + upsert + int** (≈1 dia): `speechImport.ts`, `speechClassifier.ts`, `scripts/import-camara-speeches.mjs`, `package.json`. Int verde: upsert duas vezes = 1 speech + N segmentos estáveis; `classifiedBy='manual'` preservado por upsert sem `req.user`; `segments: undefined` não apaga; busca `contains` normalizada acha acento; estado de skip bate por `audioId+excerptTMs`. Smoke real com `--legislature 57 --limit 3` (com e sem `--skip-transcribe`).
4. **Prova viva 57ª** (≈0,5 dia de operação; ≈2–3h de máquina, sem trabalho humano): `pnpm camara:import --legislature 57` em `teqo_wt153`; depois **re-executar** e conferir `created=0`/sem duplicata; verificar contagens do relatório (234 listados, `%` com segmentos, falhas), busca por "saude", "SUS", "Petrobras" e filtro por município citado (ex.: Feira de Santana) via Local API; guardar o JSON do relatório.
5. **Gates / changelog / PR** (≈0,25 dia): `pnpm gate:fast`, `pnpm migrate` + `migrate:status` limpos, `pnpm knip`, `pnpm generate:types` sem diff, `pnpm test:e2e:affected` (prefixo `src/utilities/access` → curated `campaignPermissionProfileHttp`/`campaignDemandVisibility`), changelog, `pnpm push`; PR ready (merge pelo fluxo OPS71). C155 fica fora.
**Quota total:** ~2,75 dias, dentro do appetite; o caminho crítico é a Fase 4 (wall-clock), que roda desatendida.
## Rabbit holes / Não escopo (engenharia)
- **Knip e dynamic import:** o script importa `src/` por dynamic import literal (como `seed-tse-results.mjs:44-50`); não usar `import(variable)` nem `require` — quebra o grafo do knip (`exports: error`). Testes unit/int importam os módulos estaticamente e ancoram o resto.
- **Top-level pin de `src/utilities`:** tudo em `src/utilities/speech/`; nada solto na raiz de `utilities/`.
- **Editor de vídeo / clipes / espelhamento S3:** fora (C155/fase própria); só links do VOD e cache local gitignored.
- **Comissões e audiências:** a API de discursos cobre Plenário; fora.
- **Normalizar pessoas citadas como `Contact`:** fora; menções são texto pesquisável.
- **54ª legislatura (2011–2015):** sem trecho por decisão; tratar como vazia, não como falha.
- **UI/busca do assessor (C154):** aqui só o contrato (constantes de faceta, `normalizeForSearch`, shape da query).
- **Reclassificação automática agendada / backfill 2011–2026 / produção:** C155.
- **Busca semântica, `unaccent`, `tsvector`, query builder compartilhado:** fora (D4); se o C154 precisar de mais que `searchText contains normalizeForSearch(q)`, revisitar lá.
- **Collection `person` paralela / Consent novo:** dado parlamentar público (CC BY 4.0), sem PII de cidadão; o guardrail é access, não Consent.
- **Rate-limit scheduler / fila:** o run é sequencial e resumível; nada de job queue.
## Riscos e mitigação
- **Anti-crawl (400 a UA de curl):** UA de browser do `CAMARA_USER_AGENT`; nunca bater na URL de trecho com `crawl=no`; só página do evento + `video-sob-demanda`; cache por evento em `data/camara/events/`; delay educado e retry com backoff.
- **VOD `INDISPONIVEL` ou preso em `GERANDO`:** `parseVodStatus` cobre; fallback YouTube (`urlRegistro` + offset `t/1000 − startTime`); sem `urlRegistro` → "sem vídeo" no relatório; nunca falha o import.
- **54ª/sessão sem âncora:** "sem trecho" documentado (C152); metadados entram, ASR não; cobertura útil 2015+.
- **LLM sem chave/falha:** fallback gazetteer-only (`classifiedBy='gazetteer'`, scopes vazios); `--reclassify` reprocessa depois sem re-transcrever.
- **Custo/tempo do run longo:** ~15–16h de áudio ≈ US$ 0,42 de ASR + ~US$ 0,02 de LLM + 2–3h de parede; script resumível (`sourceKey` + cache de MP4/página), `--limit` para smoke, falha por discurso não aborta o run.
- **Enum ADD VALUE em transação:** PG 12+ permite, mas o valor não pode ser usado na mesma transação; a migration só adiciona (`IF NOT EXISTS`) e nada insere `communicator` nela; `down` irreversível documentado (precedente do remodel).
- **Layout da página do evento mudar:** parsers devolvem vazio/null, nunca lançam; unit com HTML congelado detecta a quebra; relatório expõe `withoutExcerpt`/`presidingOfficer=null`.
- **Rate limit/instabilidade da API:** sequencial, `itens=100`, retry limitado com backoff (padrão do pilot), cache agressivo; falha vira `failures[]` e retoma na próxima execução.
- **Colisão de `sourceKey`** (dois itens com mesmo `dataHoraInicio|tipo|fase`): detectada no run, segundo item vira falha reportada (nunca sobrescreve em silêncio); se ocorrer, revisitar a chave incluindo `urlTexto`.
- **Faceta manual sobrescrita:** importer pula campos quando existente é `manual` + hook de preservação para escrita sem `req.user`; int test prova.
## Aceite de engenharia
- [ ] Aceite de produto da intenção coberto: 57ª no banco local com transcrição oficial + segmentos + facetas + menções; reexecução sem duplicar; relatório por execução (total, com vídeo, sem trecho, falhas, tempo, custo ASR); busca por palavra com e sem acento; keywords cruas preservadas.
- [ ] Invariantes AGENTS/engineering-standards: `access` explícito nas duas collections; migration via `pnpm migrate:create` (+ índice hand-written registrado); `src/utilities/speech/`; `server-only` nos módulos acoplados; identificadores em inglês; sem `Contact` paralelo; `data/camara/` gitignored; `overrideAccess: true` com comentário "bypass"; knip sem órfão; crédito CC BY 4.0 no admin/relatório.
- [ ] Testes de domínio: unit (facets, search, presiding, role predicate) e int (access allow/deny, upsert idempotente, replace de segmentos, manual preservado, `segments: undefined`, hook `searchText` + `contains`).
- [ ] Gates: `pnpm gate:fast`; `pnpm migrate`/`migrate:status` limpos; `pnpm knip`; `pnpm generate:types` sem diff; `pnpm test:e2e:affected`; `pnpm push`; changelog `docs/changelog/2026-09-12-c153.md`.
- [ ] Prova viva registrada (relatório JSON da 57ª + re-run idempotente) no PR.
## Self-score decision-quality
1. **Decisões caras têm rejeitadas?** 5/5 — as seis decisões caras (schema, unicidade/idempotência, provedor LLM, mecanismo de busca, RBAC, dado de "quem presidia") estão no formato Opções|Recomendação|Rejeitadas, com o porquê das rejeitadas; as baratas (nomes de campos, ordem de colunas, prompt) ficam como fill-in.
2. **Cabe no appetite?** 5/5 — ~2,75 dias contra 2–3 herdados; o único trecho longo é wall-clock desatendido (Fase 4), sem UI e sem pipeline de mídia.
3. **Rabbit holes nomeados?** 5/5 — os quatro de produto da intenção + os de engenharia (knip/dynamic import, top-level pin, unaccent/tsvector, query builder, job queue, collection de pessoa).
4. **Depth check reusa?** 5/5 — `withPayloadTransaction`, `payloadAdminOnly`/padrão de `access/*` + barrel, `pg_trgm` hand-written, `seed-tse-results.mjs` como molde de CLI, `SupporterImportBatch` como precedente de admin-hidden, `municipalityCatalog` como gazetteer, e o `camaraSpeeches.mjs` **estendido** (não duplicado) para a mesa. Nenhum pass-through novo.
5. **Intenção permanece satisfeita?** 5/5 — a engenharia não reescreveu o outcome: sem UI, sem espelho, sem `Contact`, leitura restrita com papel novo que não herda staff, e a única reavaliação (D3) entrega a mesma opção A da intenção por uma rota que já tem chave, registrada para o gate.
## Débitos (triage pós-simplify)
Nenhum achado dos dois revisores virou Issue nova (nenhum `expensive_lock` com score ≥4). O que ficou:
**Deferido com gatilho:**
- **Cache/paginação por script** (`scripts/import-camara-speeches.mjs` vs `pilot-camara-speeches.mjs`): a extração parou no HTTP (`scripts/lib/camaraFetch.mjs`); o fluxo de cache de evento/paginação vive só no import. Gatilho: o pilot virar recorrente ou um 3º consumidor do fluxo.
- **Constantes Deep Infra duplicadas** (`scripts/lib/camaraSpeeches.mjs` vs `src/utilities/ai/deepInfraTranscribe.ts`): o pilot roda em node puro e não importa `src/`. Gatilho: mudança de endpoint/modelo ou 3º consumidor.
- **`llm.calls` conta tentativas** mesmo sem `DEEPINFRA_API_KEY` (o import degrada para gazetteer por design; o pilot faz preflight e morre). Gatilho: próximo ajuste do relatório de import.
- **Índice em `speechSegment.order`**: YAGNI (poucos segmentos por discurso; a FK `speech` já filtra). Gatilho: EXPLAIN/latência real no C154.
- **Chips do Sollinha para `communicator`** (`src/lib/sollinhaOpeningQuestions.ts`): o fallback não-staff devolve chips de líder; inofensivo (access é checado no servidor), mas fora do vertical. Gatilho: C154, dono da vertical de comunicação.
**Descartado:** redundância do `searchText` no create (exigida pelo `required` do tipo gerado); ordenação defensiva dupla do presiding officer; comentários "Intentional bypass" por site (exigidos pelo ratchet); wall-clock do run (resumível/idempotente).
**Já resolvido no simplify (não reabrir):**
- Perda de mídia quando o evento não resolve no run: `excerptChanged` só compara com excerpt presente e `preserveStoredMedia` mantém `audioId`/`excerptTMs`/VOD/duração (bug HIGH do revisor de qualidade).
- `--reclassify` com LLM fora do ar não rebaixa mais uma classificação `llm` para gazetteer (preserva people/programs/projects).
- Envelope do provider validado por zod (sem `as` no `speechClassifier`).
- `parseDurationToSeconds` estrito (rejeita `1::30`, `1:99:99`, `0h99'00"`).
- Testes unit dos helpers puros (`speechSourceKey`, `parseOfficialKeywords`, `legislatureForDate`, `parseDurationToSeconds`).
- Léxico gazetteer separado em `src/lib/speechGazetteer.ts` (o contrato `speechFacets.ts` fica leve para o bundle do C154).
- Aliases de label do `topicByKey` derivados de `SPEECH_TOPICS` (fim da lista manual).
- Guard de `year` no import (data malformada não vira 0).

View file

@ -82,6 +82,7 @@
"images:resize": "node scripts/resize-images.mjs",
"images:fill": "node scripts/fill-image.mjs",
"camara:pilot": "node scripts/pilot-camara-speeches.mjs",
"camara:import": "cross-env NODE_OPTIONS=\"--no-deprecation --import=tsx/esm --import=./scripts/seed-loader.mjs\" node scripts/import-camara-speeches.mjs",
"typecheck": "tsc --noEmit",
"typecheck:tsc6": "tsc6 --noEmit",
"prepare": "husky",

View file

@ -0,0 +1,510 @@
/**
* C153 import — populates the speech catalog from the Câmara sources:
* open-data speeches → event on that date → speaker excerpt on the event page
* → async VOD → Deep Infra Whisper segments (minutagem) + facet classification
* (gazetteer + LLM) → idempotent upsert into `speech`/`speechSegment`.
*
* Re-running is safe: records are matched by `sourceKey`, metadata is
* refreshed, VOD/ASR is skipped when the excerpt identity and segments are
* already stored, and a `manual` facet curation is never overwritten.
*
* Media and reports land under `data/camara/` (gitignored). The local-DB guard
* refuses a non-local DATABASE_URL; production is C155.
*
* Usage:
* pnpm camara:import --legislature 57
* pnpm camara:import --date 2023-02-07 --limit 3
* pnpm camara:import --legislature 57 --skip-transcribe
* pnpm camara:import --legislature 57 --reclassify
*/
import { mkdir, readFile, writeFile } from 'node:fs/promises'
import { dirname, join } from 'node:path'
import { getPayload } from 'payload'
import { assertLocalDatabase } from './assert-local-database.mjs'
import {
DEEPINFRA_COST_PER_MINUTE_USD,
downloadToBuffer,
eventPageUrl,
eventsUrl,
getJson,
getText,
resolveVod,
speechesUrl,
transcribeSpeechAudio,
} from './lib/camaraFetch.mjs'
import {
LEGISLATURE_RANGES,
SOLLA_DEPUTY_ID,
legislatureForDate,
matchExcerpt,
parseDurationToSeconds,
parseEventExcerpts,
parseOfficialKeywords,
parsePresidingOfficerTransitions,
resolvePresidingOfficer,
selectSpeechEvents,
speechDate,
speechSourceKey,
} from './lib/camaraSpeeches.mjs'
import { dieWithLabel, ensureCachedDownload, loadCliEnv } from './lib/cli.mjs'
loadCliEnv()
const die = dieWithLabel('camara:import')
const config = (await import('../src/payload.config.ts')).default
const { findSpeechImportState, upsertSpeechBundle } =
await import('../src/utilities/speech/speechImport.ts')
const { classifySpeech } = await import('../src/utilities/speech/speechClassifier.ts')
const DEFAULT_OUT_DIR = 'data/camara'
const MAX_EVENT_CANDIDATES = 5
const EVENT_TYPE_PRIORITY = [
'Sessão Deliberativa',
'Breves Comunicações',
'Sessão Não Deliberativa',
'Sessão Solene',
'Outro Evento',
]
// ---------------------------------------------------------------------------
// CLI
// ---------------------------------------------------------------------------
const parseArgs = (argv) => {
const options = {
legislature: 57,
date: null,
limit: null,
speaker: 'Jorge Solla',
out: DEFAULT_OUT_DIR,
skipTranscribe: false,
reclassify: false,
help: false,
}
for (let index = 0; index < argv.length; index += 1) {
const arg = argv[index]
const value = () => {
const next = argv[++index]
if (next === undefined || next.startsWith('--')) die(`faltou valor para ${arg}.`)
return next
}
if (arg === '--help' || arg === '-h') options.help = true
else if (arg === '--skip-transcribe') options.skipTranscribe = true
else if (arg === '--reclassify') options.reclassify = true
else if (arg === '--legislature') options.legislature = Number(value())
else if (arg === '--date') options.date = value()
else if (arg === '--limit') options.limit = Number(value())
else if (arg === '--speaker') options.speaker = value()
else if (arg === '--out') options.out = value()
else die(`argumento desconhecido: ${arg}`)
}
if (!options.help && !options.date && !LEGISLATURE_RANGES[options.legislature]) {
die('informe --legislature <54|55|56|57> ou --date <YYYY-MM-DD> (veja --help).')
}
if (!options.help && options.date && !/^\d{4}-\d{2}-\d{2}$/.test(options.date)) {
die('--date deve ser YYYY-MM-DD.')
}
if (options.limit !== null && (!Number.isInteger(options.limit) || options.limit < 1)) {
die('--limit deve ser >= 1.')
}
if (!options.help && options.out.includes('..')) {
die('--out não pode escapar do diretório do repo (sem "..").')
}
return options
}
const HELP = `
Uso: pnpm camara:import [--legislature 54|55|56|57] [--date YYYY-MM-DD] [opções]
Opções:
--legislature <n> 54|55|56|57 (default 57; ignorado se --date)
--date <YYYY-MM-DD> discursos de um único dia
--limit <n> quantos discursos processar (default: todos)
--speaker <nome> nome do orador a casar no evento (default "Jorge Solla")
--out <dir> diretório de cache/artefatos (default data/camara)
--skip-transcribe não chama a Deep Infra (preserva os segmentos existentes)
--reclassify refaz a classificação mesmo já tendo facetas da LLM
--help esta ajuda
A página do evento é cacheada em <out>/events; apague-a para reler a Câmara.
`
// ---------------------------------------------------------------------------
// Caches
// ---------------------------------------------------------------------------
const eventsByDate = new Map()
const eventPages = new Map()
async function getEventsForDate(date) {
const cached = eventsByDate.get(date)
if (cached) return cached
const events = (await getJson(eventsUrl(date))).dados ?? []
eventsByDate.set(date, events)
return events
}
async function loadEventPage(eventId, options) {
const cached = eventPages.get(eventId)
if (cached) return cached
const path = join(options.out, 'events', `${eventId}.html`)
let html
try {
html = await readFile(path, 'utf8')
} catch (error) {
if (error?.code !== 'ENOENT') throw error
html = await getText(eventPageUrl(eventId), { timeoutMs: 60_000 })
await mkdir(dirname(path), { recursive: true })
await writeFile(path, html)
}
eventPages.set(eventId, html)
return html
}
async function fetchSpeeches(deputyId, from, to) {
const speeches = []
for (let page = 1; ; page += 1) {
const body = await getJson(`${speechesUrl(deputyId, from, to)}&pagina=${page}`)
const batch = body.dados ?? []
speeches.push(...batch)
const hasNext = (body.links ?? []).some((link) => link?.rel === 'next')
if (!hasNext || batch.length === 0) break
}
return speeches
}
// ---------------------------------------------------------------------------
// Pipeline
// ---------------------------------------------------------------------------
const eventPriority = (event) => {
const index = EVENT_TYPE_PRIORITY.indexOf(event?.descricaoTipo)
return index === -1 ? EVENT_TYPE_PRIORITY.length : index
}
/** Nearest event of the speech date, preferred session types first. */
async function resolveSpeechContext(speech, options) {
const candidates = selectSpeechEvents(
await getEventsForDate(speechDate(speech.dataHoraInicio)),
speech.dataHoraInicio,
)
const ordered = [...candidates].sort((left, right) => eventPriority(left) - eventPriority(right))
for (const event of ordered.slice(0, MAX_EVENT_CANDIDATES)) {
const html = await loadEventPage(event.id, options)
const excerpt = matchExcerpt(
parseEventExcerpts(html, event.id),
speech.dataHoraInicio,
options.speaker,
)
if (excerpt) return { event, html, excerpt }
}
return { event: ordered[0] ?? null, html: null, excerpt: null }
}
async function transcribeExcerpt(eventId, excerpt, options) {
const vod = await resolveVod(eventId, excerpt)
if (vod.status.state !== 'PRONTO' || !vod.status.video?.downloadUrl) {
return {
vodState: vod.status.state,
failure: `VOD não ficou PRONTO (estado=${vod.status.state})`,
}
}
const cached = await ensureCachedDownload({
label: 'camara:import',
key: `${eventId}-${excerpt.audioId}-${excerpt.tMs}`,
url: vod.status.video.downloadUrl,
ext: 'mp4',
cacheDir: options.out,
download: downloadToBuffer,
})
const { transcription, elapsedMs } = await transcribeSpeechAudio(cached.buffer)
return {
vodState: vod.status.state,
segments: transcription.segments.map((segment) => ({
startSeconds: segment.start,
endSeconds: segment.end,
text: segment.text,
})),
durationSeconds: transcription.duration,
playbackUrl: vod.status.video.playbackUrl,
downloadUrl: vod.status.video.downloadUrl,
audioSeconds: transcription.duration ?? 0,
elapsedMs,
}
}
async function processSpeech(payload, speech, options, run) {
const sourceKey = speechSourceKey(speech)
const report = {
sourceKey,
speechAt: speech.dataHoraInicio,
eventId: null,
excerpt: null,
vodState: null,
presidingOfficer: null,
classifiedBy: null,
segmentCount: 0,
created: false,
manualFacetsPreserved: false,
failures: [],
}
const existing = await findSpeechImportState(payload, sourceKey)
const keywords = parseOfficialKeywords(speech.keywords)
const date = speechDate(speech.dataHoraInicio)
const year = /^\d{4}-\d{2}-\d{2}$/.test(date) ? Number(date.slice(0, 4)) : null
let context = { event: null, html: null, excerpt: null }
try {
context = await resolveSpeechContext(speech, options)
} catch (error) {
report.failures.push({ stage: 'event', message: error?.message ?? String(error) })
}
const { event, excerpt } = context
if (event) report.eventId = event.id
if (excerpt) {
report.excerpt = { audioId: excerpt.audioId, tMs: excerpt.tMs, startTime: excerpt.startTime }
}
const transitions = context.html ? parsePresidingOfficerTransitions(context.html) : []
report.presidingOfficer = excerpt ? resolvePresidingOfficer(transitions, excerpt.tMs) : null
// Identity/media carry-over: a run that failed to resolve the event page must
// not wipe the stored excerpt or force a re-transcription next time.
const excerptChanged =
Boolean(excerpt) &&
Boolean(existing) &&
(existing.audioId !== excerpt.audioId || existing.excerptTMs !== excerpt.tMs)
const preserveStoredMedia = !excerpt || !excerptChanged
const needsTranscription =
Boolean(excerpt) &&
!options.skipTranscribe &&
(!existing || excerptChanged || existing.segmentCount === 0)
let media = {}
if (needsTranscription) {
try {
media = await transcribeExcerpt(event.id, excerpt, options)
if (media.failure) report.failures.push({ stage: 'vod', message: media.failure })
if (media.segments) {
run.asr.calls += 1
run.asr.audioSeconds += media.audioSeconds
run.asr.elapsedMs += media.elapsedMs ?? 0
}
} catch (error) {
run.asr.failures += 1
report.failures.push({ stage: 'asr', message: error?.message ?? String(error) })
}
}
const skipClassification =
existing?.classifiedBy === 'manual' || (existing?.classifiedBy === 'llm' && !options.reclassify)
let facets
if (skipClassification) {
report.classifiedBy = existing.classifiedBy
} else {
const classification = await classifySpeech({
transcript: speech.transcricao ?? '',
summary: speech.sumario ?? null,
keywords,
})
run.llm.calls += 1
if (classification.llm.used) {
facets = classification.facets
report.classifiedBy = classification.facets.classifiedBy
run.llm.used += 1
run.llm.totalTokens += classification.llm.totalTokens ?? 0
run.llm.estimatedCostUsd += classification.llm.estimatedCostUsd ?? 0
} else if (existing?.classifiedBy === 'llm') {
// Never downgrade a stored LLM classification to the gazetteer fallback
// (that would drop the extracted people/programs/projects).
report.classifiedBy = existing.classifiedBy
report.failures.push({
stage: 'classify',
message: `LLM indisponível (${classification.llm.error ?? 'erro'}); facetas anteriores preservadas`,
})
run.llm.failed += 1
} else {
facets = classification.facets
report.classifiedBy = classification.facets.classifiedBy
run.llm.failed += 1
if (classification.llm.error && run.llm.sampleError === null) {
run.llm.sampleError = classification.llm.error
}
}
}
const durationSeconds =
media.durationSeconds ??
(preserveStoredMedia ? (existing?.durationSeconds ?? null) : null) ??
parseDurationToSeconds(excerpt?.duration)
const bundle = {
sourceKey,
speechAt: String(speech.dataHoraInicio ?? '').trim(),
year,
legislature: legislatureForDate(speech.dataHoraInicio),
type: speech.tipoDiscurso ?? null,
phase: speech.faseEvento?.titulo ?? null,
durationSeconds,
summary: speech.sumario ?? null,
officialTranscript: speech.transcricao ?? null,
officialTextUrl: speech.urlTexto ?? null,
keywords,
eventId: event?.id ?? null,
eventType: event?.descricaoTipo ?? null,
eventStartAt: event?.dataHoraInicio ?? null,
eventEndAt: event?.dataHoraFim ?? null,
youtubeUrl: event?.urlRegistro ?? null,
presidingOfficer: report.presidingOfficer,
audioId: excerpt?.audioId ?? (preserveStoredMedia ? (existing?.audioId ?? null) : null),
excerptTMs: excerpt?.tMs ?? (preserveStoredMedia ? (existing?.excerptTMs ?? null) : null),
vodPlaybackUrl:
media.playbackUrl ?? (preserveStoredMedia ? (existing?.vodPlaybackUrl ?? null) : null),
vodDownloadUrl:
media.downloadUrl ?? (preserveStoredMedia ? (existing?.vodDownloadUrl ?? null) : null),
facets,
segments: media.segments,
}
try {
const counts = await upsertSpeechBundle(payload, bundle)
report.created = counts.created
report.manualFacetsPreserved = counts.manualFacetsPreserved
report.segmentCount = media.segments ? media.segments.length : (existing?.segmentCount ?? 0)
run.totals.created += counts.created ? 1 : 0
run.totals.updated += counts.created ? 0 : 1
} catch (error) {
report.failures.push({ stage: 'upsert', message: error?.message ?? String(error) })
}
run.totals.withExcerpt += excerpt ? 1 : 0
run.totals.withoutExcerpt += excerpt ? 0 : 1
run.totals.withSegments += report.segmentCount > 0 ? 1 : 0
run.totals.failed += report.failures.length > 0 ? 1 : 0
return report
}
// ---------------------------------------------------------------------------
// Report
// ---------------------------------------------------------------------------
const printReport = (run) => {
const { totals, asr, llm, failures } = run
console.log('\n=== Relatório do import C153 ===')
console.log(
`discursos: ${totals.listed} listados, ${totals.processed} processados ` +
`(${totals.created} criados, ${totals.updated} atualizados)`,
)
console.log(
`vídeo: ${totals.withExcerpt} com trecho, ${totals.withoutExcerpt} sem trecho; ` +
`${totals.withSegments} com segmentos`,
)
console.log(
`ASR: ${asr.calls} chamadas, ${(asr.audioSeconds / 60).toFixed(1)} min de áudio, ` +
`~US$ ${((asr.audioSeconds * DEEPINFRA_COST_PER_MINUTE_USD) / 60).toFixed(4)}, ` +
`${(asr.elapsedMs / 1000).toFixed(0)}s de processamento, ${asr.failures} falhas`,
)
console.log(
`LLM: ${llm.used}/${llm.calls} usadas, ${llm.totalTokens} tokens, ` +
`~US$ ${llm.estimatedCostUsd.toFixed(4)}${llm.failed > 0 ? `, ${llm.failed} falhas` : ''}`,
)
console.log(`tempo total: ${(run.elapsedMs / 1000).toFixed(1)}s; falhas: ${failures.length}`)
for (const failure of failures) {
console.log(` ✗ ${failure.speechAt} [${failure.stage}] ${failure.message}`)
}
}
const speechLine = (report) => {
const parts = [
report.created ? 'criado' : 'atualizado',
report.excerpt ? `trecho a=${report.excerpt.audioId} t=${report.excerpt.tMs}` : 'sem trecho',
]
if (report.vodState) parts.push(`VOD ${report.vodState}`)
if (report.segmentCount > 0) parts.push(`${report.segmentCount} segmentos`)
if (report.presidingOfficer) parts.push(`presidiu ${report.presidingOfficer}`)
if (report.classifiedBy) parts.push(`facetas ${report.classifiedBy}`)
if (report.failures.length > 0) parts.push(`${report.failures.length} falha(s)`)
return parts.join(' · ')
}
// ---------------------------------------------------------------------------
// Main
// ---------------------------------------------------------------------------
async function main() {
const options = parseArgs(process.argv.slice(2))
if (options.help) {
console.log(HELP)
process.exit(0)
}
assertLocalDatabase('camara:import', 'O import C153 escreve no banco local; produção é C155.')
const payload = await getPayload({ config })
const from = options.date ?? LEGISLATURE_RANGES[options.legislature][0]
const to = options.date ?? LEGISLATURE_RANGES[options.legislature][1]
const startedAt = Date.now()
console.log(
`[camara:import] deputado ${SOLLA_DEPUTY_ID} | ${from}..${to} | speaker "${options.speaker}"`,
)
const speeches = await fetchSpeeches(SOLLA_DEPUTY_ID, from, to)
const targets = options.limit === null ? speeches : speeches.slice(0, options.limit)
console.log(`[camara:import] ${speeches.length} discursos na API; processando ${targets.length}`)
if (targets.length === 0) die(`nenhum discurso em ${from}..${to}.`)
const run = {
runAt: new Date().toISOString(),
range: { from, to },
options: {
legislature: options.date ? null : options.legislature,
date: options.date,
limit: options.limit,
speaker: options.speaker,
skipTranscribe: options.skipTranscribe,
reclassify: options.reclassify,
},
totals: {
listed: speeches.length,
processed: 0,
created: 0,
updated: 0,
withExcerpt: 0,
withoutExcerpt: 0,
withSegments: 0,
failed: 0,
},
asr: { calls: 0, audioSeconds: 0, elapsedMs: 0, failures: 0 },
llm: { calls: 0, used: 0, failed: 0, totalTokens: 0, estimatedCostUsd: 0, sampleError: null },
elapsedMs: 0,
speeches: [],
failures: [],
}
for (const [index, speech] of targets.entries()) {
const report = await processSpeech(payload, speech, options, run)
run.totals.processed += 1
run.speeches.push(report)
for (const failure of report.failures) {
run.failures.push({ sourceKey: report.sourceKey, speechAt: report.speechAt, ...failure })
}
console.log(`[${index + 1}/${targets.length}] ${report.speechAt} — ${speechLine(report)}`)
}
run.elapsedMs = Date.now() - startedAt
printReport(run)
const reportPath = join(options.out, 'reports', `import-${run.runAt.replace(/[:.]/g, '-')}.json`)
await mkdir(dirname(reportPath), { recursive: true })
await writeFile(reportPath, JSON.stringify(run, null, 2))
console.log(`\n[camara:import] relatório JSON: ${reportPath}`)
process.exit(0)
}
main().catch((error) => {
console.error(error)
die(error?.message || String(error))
})

119
scripts/lib/camaraFetch.mjs Normal file
View file

@ -0,0 +1,119 @@
/**
* HTTP surface shared by the Câmara speech scripts (C152 pilot + C153 import):
* the open-data endpoints, the anti-crawl browser UA, bounded retry, the async
* VOD poll and the Deep Infra transcription call. Kept apart from
* `camaraSpeeches.mjs` (pure) so the fragile rules stay unit-testable without
* network.
*/
import {
buildVodUrl,
CAMARA_USER_AGENT,
DEEPINFRA_TRANSCRIBE_URL,
DEEPINFRA_WHISPER_MODEL,
normalizeTranscription,
parseVodStatus,
} from './camaraSpeeches.mjs'
const camaraHeaders = { 'User-Agent': CAMARA_USER_AGENT, Accept: 'application/json' }
const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms))
async function fetchOnce(url, { timeoutMs = 30_000, headers = camaraHeaders } = {}) {
const response = await fetch(url, { headers, signal: AbortSignal.timeout(timeoutMs) })
if (!response.ok) throw new Error(`HTTP ${response.status} em ${url}`)
return response
}
/** The Câmara API drops connections now and then — bounded retry for GETs. */
export async function getText(url, { attempts = 3, ...options } = {}) {
let lastError
for (let attempt = 1; attempt <= attempts; attempt += 1) {
try {
return await (await fetchOnce(url, options)).text()
} catch (error) {
lastError = error
if (attempt < attempts) await sleep(2_000)
}
}
throw lastError
}
export const getJson = async (url, options) => JSON.parse(await getText(url, options))
export const downloadToBuffer = async (url, { attempts = 3, timeoutMs = 120_000 } = {}) => {
let lastError
for (let attempt = 1; attempt <= attempts; attempt += 1) {
try {
const response = await fetchOnce(url, {
timeoutMs,
headers: { 'User-Agent': CAMARA_USER_AGENT },
})
return Buffer.from(await response.arrayBuffer())
} catch (error) {
lastError = error
if (attempt < attempts) await sleep(3_000)
}
}
throw lastError
}
export const speechesUrl = (deputyId, from, to) =>
`https://dadosabertos.camara.leg.br/api/v2/deputados/${deputyId}/discursos` +
`?dataInicio=${from}&dataFim=${to}&itens=100&ordenarPor=dataHoraInicio&ordem=ASC`
export const eventsUrl = (date) =>
`https://dadosabertos.camara.leg.br/api/v2/eventos?dataInicio=${date}&dataFim=${date}&itens=100`
export const eventPageUrl = (eventId) => `https://www.camara.leg.br/evento-legislativo/${eventId}`
const VOD_POLL_INTERVAL_MS = 5_000
const VOD_POLL_MAX_ATTEMPTS = 40
/** Polls the async VOD generation until PRONTO (or gives up). */
export async function resolveVod(eventId, excerpt) {
const url = buildVodUrl(eventId, excerpt.audioId, excerpt.tMs)
let lastStatus = null
for (let attempt = 1; attempt <= VOD_POLL_MAX_ATTEMPTS; attempt += 1) {
const status = parseVodStatus(await getJson(url, { timeoutMs: 60_000 }))
lastStatus = status
if (status.state === 'PRONTO' || status.state === 'INDISPONIVEL') {
return { url, status, attempts: attempt }
}
await sleep(VOD_POLL_INTERVAL_MS)
}
return { url, status: lastStatus, attempts: VOD_POLL_MAX_ATTEMPTS }
}
export const DEEPINFRA_COST_PER_MINUTE_USD = 0.00045
/**
* Sends the excerpt MP4 (video/mp4 is accepted; no ffmpeg) to Deep Infra's
* OpenAI-compatible endpoint and returns the normalized transcription.
* Throws on missing key or provider error — the caller owns the failure policy.
*/
export async function transcribeSpeechAudio(buffer) {
const apiKey = process.env.DEEPINFRA_API_KEY
if (!apiKey) {
throw new Error('DEEPINFRA_API_KEY ausente — configure no .env.local ou no env do processo.')
}
const form = new FormData()
form.append('file', new Blob([buffer], { type: 'video/mp4' }), 'trecho.mp4')
form.append('model', DEEPINFRA_WHISPER_MODEL)
form.append('language', 'pt')
form.append('response_format', 'verbose_json')
form.append('timestamp_granularities[]', 'segment')
const startedAt = Date.now()
const response = await fetch(DEEPINFRA_TRANSCRIBE_URL, {
method: 'POST',
headers: { Authorization: `Bearer ${apiKey}` },
body: form,
signal: AbortSignal.timeout(600_000),
})
if (!response.ok) {
throw new Error(`Deep Infra HTTP ${response.status}: ${(await response.text()).slice(0, 300)}`)
}
return {
transcription: normalizeTranscription(await response.json()),
elapsedMs: Date.now() - startedAt,
}
}

View file

@ -1,7 +1,9 @@
/**
* Pure derivation for the C152 pilot (`scripts/pilot-camara-speeches.mjs`):
* matching a Câmara speech to its legislative event and to the speaker's video
* excerpt, plus normalizing the VOD status and the Deep Infra transcription.
* Pure derivation for the Câmara speech scripts — the C152 pilot
* (`scripts/pilot-camara-speeches.mjs`) and the C153 catalog import
* (`scripts/import-camara-speeches.mjs`): matching a Câmara speech to its
* legislative event and to the speaker's video excerpt, the presiding-officer
* transitions, plus normalizing the VOD status and the Deep Infra transcription.
*
* No I/O here — the CLI threads fetch/download/transcribe through these
* decisions, so the fragile rules (HTML anchors, time-of-day matching, epoch
@ -73,6 +75,83 @@ export const speechTimeOfDaySeconds = (iso) => clockToSeconds(String(iso ?? '').
/** `YYYY-MM-DD` prefix of a naive datetime. */
export const speechDate = (iso) => String(iso ?? '').slice(0, 10)
/**
* Stable natural key of an open-data speech. The API has no id, so identity is
* the naive datetime plus the speech type and event phase — the record is
* updated in place across imports (C153 idempotency).
*
* @param {{ dataHoraInicio?: string, tipoDiscurso?: string, faseEvento?: { titulo?: string } } | null | undefined} speech
* @returns {string}
*/
export const speechSourceKey = (speech) =>
[
String(speech?.dataHoraInicio ?? '').trim(),
String(speech?.tipoDiscurso ?? '').trim(),
String(speech?.faseEvento?.titulo ?? '').trim(),
].join('|')
/**
* Official keywords come as a newline-separated string; keep each keyword raw.
*
* @param {unknown} raw
* @returns {string[]}
*/
export const parseOfficialKeywords = (raw) =>
String(raw ?? '')
.split(/[\r\n]+/)
.map((keyword) => keyword.trim())
.filter(Boolean)
/**
* Legislature of a `YYYY-MM-DD` date per `LEGISLATURE_RANGES`, or null when it
* falls outside the supported range.
*
* @param {string} iso
* @returns {string | null}
*/
export const legislatureForDate = (iso) => {
const date = speechDate(iso)
if (!/^\d{4}-\d{2}-\d{2}$/.test(date)) return null
for (const [legislature, [from, to]] of Object.entries(LEGISLATURE_RANGES)) {
if (date >= from && date <= to) return legislature
}
return null
}
/**
* Parses the Câmara duration spellings into seconds: `0h04'03"` (excerpt card)
* and `0:04:07` / `4:07` (VOD status). Null when unparseable.
*
* @param {unknown} value
* @returns {number | null}
*/
export const parseDurationToSeconds = (value) => {
const raw = String(value ?? '').trim()
if (raw === '') return null
const card = /^(\d+)h(\d{1,2})'(\d{1,2})"?$/.exec(raw)
if (card) {
const minutes = Number(card[2])
const seconds = Number(card[3])
if (minutes > 59 || seconds > 59) return null
return Number(card[1]) * 3600 + minutes * 60 + seconds
}
const parts = raw.split(':')
if (!parts.every((part) => /^\d{1,2}$/.test(part))) return null
if (parts.length === 3) {
const [hours, minutes, seconds] = parts.map(Number)
if (minutes > 59 || seconds > 59) return null
return hours * 3600 + minutes * 60 + seconds
}
if (parts.length === 2) {
const [minutes, seconds] = parts.map(Number)
if (seconds > 59) return null
return minutes * 60 + seconds
}
return null
}
const integerOrNull = (value) => {
const n = Number(value)
return Number.isInteger(n) && n > 0 ? n : null
@ -200,6 +279,67 @@ export const matchExcerpt = (excerpts, speechIso, speaker) => {
return bySpeaker.reduce((best, current) => (delta(current) < delta(best) ? current : best))
}
/**
* Parses the "Troca da mesa" transition groups of an event page. Each heading
* `<h4 class="g-l-assista__categoria-outros-videos">Troca da mesa Presidente X
* por Participante Y</h4>` labels the excerpt cards that follow it; the newest
* card of the group (`t` epoch ms) marks the observed moment of the change.
* Returns transitions ascending by time; `[]` when the layout changed or the
* page recorded none.
*
* @param {string} html
* @returns {Array<{ from: string, to: string, tMs: number }>}
*/
export const parsePresidingOfficerTransitions = (html) => {
const page = String(html ?? '')
const headingPattern =
/<h4 class="g-l-assista__categoria-outros-videos">Troca da mesa Presidente (.+?) por Participante (.+?)<\/h4>/g
const transitions = []
let heading
while ((heading = headingPattern.exec(page)) !== null) {
const nextHeading = page.indexOf(
'g-l-assista__categoria-outros-videos',
headingPattern.lastIndex,
)
const group = page.slice(heading.index, nextHeading === -1 ? page.length : nextHeading)
const times = [...group.matchAll(/<a id="link-trecho-video" href="[^"]*?t&#x3D;(\d+)[^"]*"/g)]
.map((match) => Number(match[1]))
.filter((value) => Number.isInteger(value) && value > 0)
if (times.length === 0) continue
transitions.push({
from: heading[1].trim(),
to: heading[2].trim(),
tMs: Math.max(...times),
})
}
return transitions.sort((left, right) => left.tMs - right.tMs)
}
/**
* Who presided at `tMs` according to the transition groups: the incoming
* officer (`to`) of the latest transition at or before the moment; before the
* first transition, the outgoing officer (`from`) of the earliest one. Returns
* null when the page recorded no transitions (unknown, never invented).
*
* @param {ReturnType<typeof parsePresidingOfficerTransitions>} transitions
* @param {number} tMs
* @returns {string | null}
*/
export const resolvePresidingOfficer = (transitions, tMs) => {
const ordered = (Array.isArray(transitions) ? transitions : [])
.filter((transition) => Number.isFinite(transition?.tMs))
.sort((left, right) => left.tMs - right.tMs)
const target = Number(tMs)
if (ordered.length === 0 || !Number.isFinite(target)) return null
let current = ordered[0].from
for (const transition of ordered) {
if (transition.tMs > target) break
current = transition.to
}
return current
}
/**
* Normalizes a Deep Infra transcription response. Accepts the OpenAI-compat
* shape (`segments[{start,end,text}]`) and the native whisper shape

View file

@ -21,16 +21,21 @@ import { access, mkdir, readFile, writeFile } from 'node:fs/promises'
import { join } from 'node:path'
import {
CAMARA_USER_AGENT,
DEEPINFRA_TRANSCRIBE_URL,
DEEPINFRA_WHISPER_MODEL,
DEEPINFRA_COST_PER_MINUTE_USD,
downloadToBuffer,
eventPageUrl,
eventsUrl,
getJson,
getText,
resolveVod,
speechesUrl,
transcribeSpeechAudio,
} from './lib/camaraFetch.mjs'
import {
LEGISLATURE_RANGES,
SOLLA_DEPUTY_ID,
buildVodUrl,
matchExcerpt,
normalizeTranscription,
parseEventExcerpts,
parseVodStatus,
selectSpeechEvents,
} from './lib/camaraSpeeches.mjs'
import { dieWithLabel, ensureCachedDownload, loadCliEnv } from './lib/cli.mjs'
@ -39,9 +44,6 @@ loadCliEnv()
const die = dieWithLabel('camara:pilot')
const DEEPINFRA_COST_PER_MINUTE_USD = 0.00045
const VOD_POLL_INTERVAL_MS = 5_000
const VOD_POLL_MAX_ATTEMPTS = 40
const DEFAULT_OUT_DIR = 'data/camara'
// ---------------------------------------------------------------------------
@ -103,49 +105,10 @@ Opções:
--help esta ajuda
`
// ---------------------------------------------------------------------------
// HTTP helpers
// ---------------------------------------------------------------------------
const camaraHeaders = { 'User-Agent': CAMARA_USER_AGENT, Accept: 'application/json' }
const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms))
async function fetchOnce(url, { timeoutMs = 30_000, headers = camaraHeaders } = {}) {
const response = await fetch(url, { headers, signal: AbortSignal.timeout(timeoutMs) })
if (!response.ok) throw new Error(`HTTP ${response.status} em ${url}`)
return response
}
/** The Câmara API drops connections now and then — bounded retry for GETs. */
async function getText(url, { attempts = 3, ...options } = {}) {
let lastError
for (let attempt = 1; attempt <= attempts; attempt += 1) {
try {
return await (await fetchOnce(url, options)).text()
} catch (error) {
lastError = error
if (attempt < attempts) await sleep(2_000)
}
}
throw lastError
}
const getJson = async (url, options) => JSON.parse(await getText(url, options))
// ---------------------------------------------------------------------------
// Pipeline steps
// ---------------------------------------------------------------------------
const speechesUrl = (deputyId, from, to) =>
`https://dadosabertos.camara.leg.br/api/v2/deputados/${deputyId}/discursos` +
`?dataInicio=${from}&dataFim=${to}&itens=100&ordenarPor=dataHoraInicio&ordem=ASC`
const eventsUrl = (date) =>
`https://dadosabertos.camara.leg.br/api/v2/eventos?dataInicio=${date}&dataFim=${date}&itens=100`
const eventPageUrl = (eventId) => `https://www.camara.leg.br/evento-legislativo/${eventId}`
const fileExists = async (path) => {
try {
await access(path)
@ -161,62 +124,6 @@ const resolveRange = (options) => {
return { from, to }
}
/** Polls the async VOD generation until PRONTO (or gives up). */
async function resolveVod(eventId, excerpt) {
const url = buildVodUrl(eventId, excerpt.audioId, excerpt.tMs)
let lastStatus = null
for (let attempt = 1; attempt <= VOD_POLL_MAX_ATTEMPTS; attempt += 1) {
const status = parseVodStatus(await getJson(url, { timeoutMs: 60_000 }))
lastStatus = status
if (status.state === 'PRONTO') return { url, status, attempts: attempt }
if (status.state === 'INDISPONIVEL') return { url, status, attempts: attempt }
await sleep(VOD_POLL_INTERVAL_MS)
}
return { url, status: lastStatus, attempts: VOD_POLL_MAX_ATTEMPTS }
}
const downloadToBuffer = async (url) => {
let lastError
for (let attempt = 1; attempt <= 3; attempt += 1) {
try {
const response = await fetchOnce(url, {
timeoutMs: 120_000,
headers: { 'User-Agent': CAMARA_USER_AGENT },
})
return Buffer.from(await response.arrayBuffer())
} catch (error) {
lastError = error
if (attempt < 3) await sleep(3_000)
}
}
throw lastError
}
async function transcribe(mp4Buffer) {
const apiKey = process.env.DEEPINFRA_API_KEY
if (!apiKey) die('DEEPINFRA_API_KEY ausente — configure no .env.local ou no env do processo.')
const form = new FormData()
form.append('file', new Blob([mp4Buffer], { type: 'video/mp4' }), 'trecho.mp4')
form.append('model', DEEPINFRA_WHISPER_MODEL)
form.append('language', 'pt')
form.append('response_format', 'verbose_json')
form.append('timestamp_granularities[]', 'segment')
const startedAt = Date.now()
const response = await fetch(DEEPINFRA_TRANSCRIBE_URL, {
method: 'POST',
headers: { Authorization: `Bearer ${apiKey}` },
body: form,
signal: AbortSignal.timeout(600_000),
})
if (!response.ok) {
throw new Error(`Deep Infra HTTP ${response.status}: ${(await response.text()).slice(0, 300)}`)
}
return {
transcription: normalizeTranscription(await response.json()),
elapsedMs: Date.now() - startedAt,
}
}
// ---------------------------------------------------------------------------
// Per-speech orchestration
// ---------------------------------------------------------------------------
@ -311,7 +218,7 @@ async function processSpeech(speech, options) {
if (!options.skipTranscribe && report.mp4) {
const buffer = await readFile(report.mp4.path)
const { transcription, elapsedMs } = await transcribe(buffer)
const { transcription, elapsedMs } = await transcribeSpeechAudio(buffer)
const estimatedCostUsd = transcription.duration
? (transcription.duration / 60) * DEEPINFRA_COST_PER_MINUTE_USD
: null
@ -396,6 +303,9 @@ async function main() {
console.log(HELP)
process.exit(0)
}
if (!options.skipTranscribe && !process.env.DEEPINFRA_API_KEY) {
die('DEEPINFRA_API_KEY ausente — configure no .env.local ou no env do processo.')
}
const { from, to } = resolveRange(options)
console.log(

View file

@ -484,6 +484,7 @@ export const CampaignUser: CollectionConfig = {
{ label: 'Coordenador Geral', value: 'coordinator' },
{ label: 'Assessor', value: 'advisor' },
{ label: 'Candidato', value: 'candidate' },
{ label: 'Assessor de Comunicação', value: 'communicator' },
{ label: 'Liderança', value: 'leader' },
],
access: {

310
src/collections/Speech.ts Normal file
View file

@ -0,0 +1,310 @@
import type {
CollectionBeforeChangeHook,
CollectionBeforeDeleteHook,
CollectionBeforeValidateHook,
CollectionConfig,
} from 'payload'
import { SPEECH_CLASSIFICATION_SOURCES, SPEECH_SCOPES, SPEECH_TOPICS } from '@/lib/speechFacets'
import { canReadSpeech, canUpdateSpeech, payloadAdminOnly } from '@/utilities/campaignAccess'
/**
* Speech catalog (C153) — one record per Jorge Solla speech in the Câmara,
* with the official transcript (taquigrafia), the ASR segments with timestamps
* (speechSegment), curated facets and extracted mentions.
*
* Data provenance: Câmara open data + VOD, CC BY 4.0 — keep the credit.
* Identity is the API speech (`sourceKey`), never the regenerable VOD URL.
*/
const FACET_FIELDS = [
'topics',
'scopes',
'classifiedBy',
'mentionedMunicipalities',
'mentionedPeople',
'mentionedPrograms',
'mentionedProjects',
] as const
const deriveSpeechYear: CollectionBeforeValidateHook = ({ data, originalDoc }) => {
if (!data) return data
const speechAt = typeof data.speechAt === 'string' ? data.speechAt : originalDoc?.speechAt
const year = Number(String(speechAt ?? '').slice(0, 4))
if (Number.isInteger(year) && year > 1900) data.year = year
return data
}
/**
* A manual facet correction is curation: unauthenticated writes (the import
* CLI) never overwrite it. Authenticated admin edits stay free to change both
* the facets and the provenance.
*/
const preserveManualFacets: CollectionBeforeChangeHook = ({
data,
operation,
originalDoc,
req,
}) => {
if (!data || operation !== 'update' || originalDoc?.classifiedBy !== 'manual') return data
if (req.user) return data
const next = data as Record<string, unknown>
for (const field of FACET_FIELDS) {
if (field in next) next[field] = originalDoc[field]
}
return data
}
/** Segments are owned by the speech; Payload relationships do not cascade. */
const deleteSpeechSegments: CollectionBeforeDeleteHook = async ({ id, req }) => {
await req.payload.delete({
collection: 'speechSegment',
where: { speech: { equals: id } },
req,
// Intentional bypass: the cascade is owned by this hook; segment access is
// admin-only and this runs inside the authorized speech delete request.
overrideAccess: true,
})
}
export const Speech: CollectionConfig = {
slug: 'speech',
labels: {
singular: 'Discurso',
plural: 'Discursos',
},
admin: {
group: 'Comunicação',
useAsTitle: 'speechAt',
defaultColumns: ['speechAt', 'type', 'phase', 'topics', 'classifiedBy'],
description:
'Acervo de falas do deputado na Câmara. Dados e vídeos da Câmara dos Deputados (CC BY 4.0).',
},
access: {
create: payloadAdminOnly,
read: canReadSpeech,
update: canUpdateSpeech,
delete: payloadAdminOnly,
},
hooks: {
beforeValidate: [deriveSpeechYear],
beforeChange: [preserveManualFacets],
beforeDelete: [deleteSpeechSegments],
},
fields: [
{
name: 'sourceKey',
type: 'text',
label: 'Chave da fonte',
required: true,
unique: true,
index: true,
admin: {
readOnly: true,
description: 'Identidade da fala na API da Câmara (data/hora + tipo + fase).',
},
},
{
// Wall-clock time of the source (Brasília local, no timezone). Kept as
// text on purpose: the Câmara API has no timezone and pretending it is an
// instant would shift old DST dates.
name: 'speechAt',
type: 'text',
label: 'Data e hora',
required: true,
index: true,
admin: {
readOnly: true,
description: 'Horário local de Brasília, como publicado pela Câmara.',
},
},
{
name: 'year',
type: 'number',
label: 'Ano',
index: true,
admin: { readOnly: true },
},
{
name: 'legislature',
type: 'select',
label: 'Legislatura',
index: true,
options: [
{ label: '54ª', value: '54' },
{ label: '55ª', value: '55' },
{ label: '56ª', value: '56' },
{ label: '57ª', value: '57' },
],
},
{
name: 'type',
type: 'text',
label: 'Tipo de discurso',
admin: { readOnly: true },
},
{
name: 'phase',
type: 'text',
label: 'Fase da sessão',
index: true,
admin: { readOnly: true },
},
{
name: 'durationSeconds',
type: 'number',
label: 'Duração (s)',
admin: { readOnly: true },
},
{
name: 'summary',
type: 'textarea',
label: 'Sumário oficial',
admin: { readOnly: true },
},
{
name: 'officialTranscript',
type: 'textarea',
label: 'Transcrição oficial',
admin: { readOnly: true },
},
{
name: 'officialTextUrl',
type: 'text',
label: 'Texto oficial (Diário)',
admin: { readOnly: true },
},
{
name: 'keywords',
type: 'text',
label: 'Palavras-chave oficiais',
hasMany: true,
admin: {
readOnly: true,
description: 'Preservadas cruas; as facetas não as substituem.',
},
},
{
name: 'eventId',
type: 'number',
label: 'Evento',
index: true,
admin: { readOnly: true },
},
{
name: 'eventType',
type: 'text',
label: 'Tipo de sessão',
admin: { readOnly: true },
},
{
name: 'eventStartAt',
type: 'text',
label: 'Início da sessão',
admin: { readOnly: true },
},
{
name: 'eventEndAt',
type: 'text',
label: 'Fim da sessão',
admin: { readOnly: true },
},
{
name: 'youtubeUrl',
type: 'text',
label: 'YouTube da sessão',
admin: { readOnly: true },
},
{
name: 'presidingOfficer',
type: 'text',
label: 'Quem presidia',
admin: {
readOnly: true,
description: 'Derivado das trocas de mesa da sessão; vazio quando não há registro.',
},
},
{
name: 'audioId',
type: 'number',
label: 'Áudio',
index: true,
admin: { readOnly: true },
},
{
name: 'excerptTMs',
type: 'number',
label: 'Início do trecho (ms)',
index: true,
admin: { readOnly: true },
},
{
name: 'vodPlaybackUrl',
type: 'text',
label: 'VOD (reproduzir)',
admin: { readOnly: true, description: 'Último link conhecido; regerável pelo VOD.' },
},
{
name: 'vodDownloadUrl',
type: 'text',
label: 'VOD (baixar)',
admin: { readOnly: true, description: 'Último link conhecido; regerável pelo VOD.' },
},
{
name: 'topics',
type: 'select',
label: 'Temas',
hasMany: true,
index: true,
options: SPEECH_TOPICS.map(({ value, label }) => ({ value, label })),
},
{
name: 'scopes',
type: 'select',
label: 'Alcance',
hasMany: true,
index: true,
options: SPEECH_SCOPES.map(({ value, label }) => ({ value, label })),
},
{
name: 'classifiedBy',
type: 'select',
label: 'Proveniência da classificação',
required: true,
defaultValue: 'gazetteer',
index: true,
options: SPEECH_CLASSIFICATION_SOURCES.map((value) => ({
value,
label: value === 'gazetteer' ? 'Gazetteer' : value === 'llm' ? 'LLM' : 'Manual',
})),
},
{
name: 'mentionedMunicipalities',
type: 'relationship',
relationTo: 'municipality',
label: 'Municípios citados',
hasMany: true,
index: true,
},
{
name: 'mentionedPeople',
type: 'text',
label: 'Pessoas citadas',
hasMany: true,
admin: { description: 'Menções textuais — não é cadastro de contatos.' },
},
{
name: 'mentionedPrograms',
type: 'text',
label: 'Programas citados',
hasMany: true,
},
{
name: 'mentionedProjects',
type: 'text',
label: 'Projetos citados',
hasMany: true,
},
],
}

View file

@ -0,0 +1,84 @@
import type { CollectionBeforeValidateHook, CollectionConfig } from 'payload'
import { normalizeForSearch } from '@/lib/speechSearch'
import { canReadSpeech, payloadAdminOnly } from '@/utilities/campaignAccess'
/**
* Searchable ASR segment of a speech (C153). The official transcript is the
* reading surface; these segments carry the timestamps and the trigram-search
* index (`speech_segment_search_text_trgm_idx`, hand-written migration) used
* by the C154 search. Admin-hidden: the product surface is the catalog UI.
*/
const deriveSearchText: CollectionBeforeValidateHook = ({ data, originalDoc }) => {
if (!data) return data
const text = typeof data.text === 'string' ? data.text : originalDoc?.text
data.searchText = normalizeForSearch(text ?? '')
return data
}
export const SpeechSegment: CollectionConfig = {
slug: 'speechSegment',
labels: {
singular: 'Segmento de fala',
plural: 'Segmentos de fala',
},
admin: {
group: 'Comunicação',
hidden: () => true,
useAsTitle: 'text',
},
access: {
create: payloadAdminOnly,
read: canReadSpeech,
update: payloadAdminOnly,
delete: payloadAdminOnly,
},
hooks: {
beforeValidate: [deriveSearchText],
},
fields: [
{
name: 'speech',
type: 'relationship',
relationTo: 'speech',
label: 'Discurso',
required: true,
index: true,
},
{
name: 'order',
type: 'number',
label: 'Ordem',
required: true,
},
{
name: 'startSeconds',
type: 'number',
label: 'Início (s)',
required: true,
},
{
name: 'endSeconds',
type: 'number',
label: 'Fim (s)',
required: true,
},
{
name: 'text',
type: 'textarea',
label: 'Texto',
required: true,
},
{
name: 'searchText',
type: 'text',
label: 'Texto normalizado (busca)',
required: true,
admin: {
readOnly: true,
description: 'Derivado do texto (sem acentos, minúsculas) para a busca por palavra.',
},
},
],
}

View file

@ -84,6 +84,9 @@ const leaderNav: CampaignNavItem[] = [
export const getCampaignNav = (role: CampaignUser['role']): CampaignNavItem[] => {
if (role === 'leader') return leaderNav
// C153 — the communication assessor is not staff: no staff destinations.
// The catalog vertical (C154) adds its own nav items here.
if (role === 'communicator') return []
return staffNav.filter((item) => {
if (item.href === '/campanha/apoiadores') return canAccessSupporterArea(role)

View file

@ -16,3 +16,11 @@ export const isStaffCampaignRole = (role: CampaignRole): boolean =>
/** Coordinator or candidate — unrestricted scope (all municipalities, decisions). */
export const isUnrestrictedCampaignRole = (role: CampaignRole): boolean =>
role === 'coordinator' || role === 'candidate'
/**
* Speech catalog readers (C153): the communication assessor plus the
* unrestricted roles. Deliberately NOT part of `isStaffCampaignRole` — the
* communicator does not inherit any staff area, only the catalog vertical.
*/
export const canReadSpeechCatalog = (role: CampaignRole): boolean =>
role === 'communicator' || isUnrestrictedCampaignRole(role)

166
src/lib/speechFacets.ts Normal file
View file

@ -0,0 +1,166 @@
import { z } from 'zod'
import type { MunicipalityCatalogEntry } from '@/lib/municipalityCatalog'
import { normalizeForSearch, uniqueByNormalizedForm } from '@/lib/speechSearch'
/**
* Facet contract for the speech catalog (C153). The topic and scope values are
* persisted data (facet filters in C154); labels are the pt-BR product names.
* The offline gazetteer lives in `speechGazetteer.ts` and the LLM refinement in
* `src/utilities/speech/speechClassifier.ts`; this module validates the LLM
* response and merges both provenances. `manual` edits win over both.
*/
export const SPEECH_TOPICS = [
{ value: 'saude', label: 'Saúde' },
{ value: 'educacao', label: 'Educação' },
{ value: 'cultura', label: 'Cultura' },
{ value: 'esporte', label: 'Esporte' },
{ value: 'seguranca-publica', label: 'Segurança Pública' },
{ value: 'meio-ambiente', label: 'Meio Ambiente' },
{ value: 'economia-trabalho', label: 'Economia e Trabalho' },
{ value: 'direitos-humanos', label: 'Direitos Humanos e Assistência Social' },
{ value: 'infraestrutura', label: 'Infraestrutura e Transporte' },
{ value: 'ciencia-tecnologia', label: 'Ciência e Tecnologia' },
{ value: 'politica-instituicoes', label: 'Política e Instituições' },
{ value: 'agricultura', label: 'Agricultura e Agropecuária' },
{ value: 'habitacao-cidades', label: 'Habitação e Cidades' },
{ value: 'comunicacao-midia', label: 'Comunicação e Mídia' },
{ value: 'igualdade-racial', label: 'Igualdade Racial' },
{ value: 'mulheres-genero', label: 'Mulheres e Gênero' },
{ value: 'juventude', label: 'Juventude' },
{ value: 'pessoa-deficiencia', label: 'Pessoa com Deficiência' },
] as const
export type SpeechTopic = (typeof SPEECH_TOPICS)[number]['value']
export const SPEECH_SCOPES = [
{ value: 'bahia', label: 'Bahia' },
{ value: 'brasil', label: 'Brasil' },
{ value: 'internacional', label: 'Internacional' },
] as const
export type SpeechScope = (typeof SPEECH_SCOPES)[number]['value']
export const SPEECH_CLASSIFICATION_SOURCES = ['gazetteer', 'llm', 'manual'] as const
type SpeechClassificationSource = (typeof SPEECH_CLASSIFICATION_SOURCES)[number]
export type SpeechFacetClassification = {
topics: SpeechTopic[]
scopes: SpeechScope[]
municipalities: MunicipalityCatalogEntry[]
people: string[]
programs: string[]
projects: string[]
}
export type MergedSpeechFacets = SpeechFacetClassification & {
classifiedBy: SpeechClassificationSource
}
const facetKey = (value: string): string => normalizeForSearch(value).replace(/-/g, ' ')
const topicByKey = new Map(
SPEECH_TOPICS.flatMap(({ value, label }) => [
[facetKey(value), value],
[facetKey(label), value],
]),
)
const scopeByKey = new Map(
SPEECH_SCOPES.flatMap(({ value, label }) => [
[facetKey(value), value],
[facetKey(label), value],
]),
)
// ---------------------------------------------------------------------------
// LLM response parsing and merge
// ---------------------------------------------------------------------------
const MAX_MENTION_ITEMS = 12
const MAX_MENTION_LENGTH = 120
const llmFacetResponseSchema = z.object({
topics: z.array(z.unknown()).optional(),
scopes: z.array(z.unknown()).optional(),
people: z.array(z.unknown()).optional(),
programs: z.array(z.unknown()).optional(),
projects: z.array(z.unknown()).optional(),
})
export type SpeechLlmFacets = {
topics: SpeechTopic[]
scopes: SpeechScope[]
people: string[]
programs: string[]
projects: string[]
}
const stringMentions = (value: unknown): string[] => {
if (!Array.isArray(value)) return []
return uniqueByNormalizedForm(
value
.filter((item): item is string => typeof item === 'string')
.map((item) => item.trim().slice(0, MAX_MENTION_LENGTH)),
).slice(0, MAX_MENTION_ITEMS)
}
/**
* Validates the LLM JSON response against the known taxonomy. Unknown topics
* and scopes are dropped (never persisted as free text); an object without any
* known key returns null so the caller falls back to the gazetteer provenance.
*/
export const parseLlmFacetResponse = (json: unknown): SpeechLlmFacets | null => {
const parsed = llmFacetResponseSchema.safeParse(json)
if (!parsed.success || Object.keys(parsed.data).length === 0) return null
const data = parsed.data
const topics = uniqueByNormalizedForm(
(Array.isArray(data.topics) ? data.topics : []).filter(
(item): item is string => typeof item === 'string',
),
)
.map((item) => topicByKey.get(facetKey(item)))
.filter((value): value is SpeechTopic => value !== undefined)
const scopes = uniqueByNormalizedForm(
(Array.isArray(data.scopes) ? data.scopes : []).filter(
(item): item is string => typeof item === 'string',
),
)
.map((item) => scopeByKey.get(facetKey(item)))
.filter((value): value is SpeechScope => value !== undefined)
return {
topics: [...new Set(topics)],
scopes: [...new Set(scopes)],
people: stringMentions(data.people),
programs: stringMentions(data.programs),
projects: stringMentions(data.projects),
}
}
/**
* Gazetteer ∪ LLM. Topics are unioned (the lexicon is evidence too); scopes
* prefer the validated LLM output when it produced any; people/programs/
* projects come from the LLM only. Provenance is `llm` whenever a valid
* response refined the result, `gazetteer` otherwise.
*/
export const mergeFacetClassification = (
gazetteer: SpeechFacetClassification,
llm: SpeechLlmFacets | null,
): MergedSpeechFacets => {
if (!llm) return { ...gazetteer, classifiedBy: 'gazetteer' }
return {
topics: [...new Set([...gazetteer.topics, ...llm.topics])],
scopes: llm.scopes.length > 0 ? llm.scopes : gazetteer.scopes,
municipalities: gazetteer.municipalities,
people: llm.people,
programs: llm.programs,
projects: llm.projects,
classifiedBy: 'llm',
}
}

443
src/lib/speechGazetteer.ts Normal file
View file

@ -0,0 +1,443 @@
import { municipalityCatalog, type MunicipalityCatalogEntry } from '@/lib/municipalityCatalog'
import {
SPEECH_SCOPES,
SPEECH_TOPICS,
type SpeechFacetClassification,
type SpeechScope,
type SpeechTopic,
} from '@/lib/speechFacets'
import { normalizeForSearch } from '@/lib/speechSearch'
/**
* Offline gazetteer pass of the speech facet classifier (C153): topic/scope
* lexicons plus municipality mentions. Deliberately a separate module from the
* `speechFacets` contract so the C154 client bundle can import the topic
* labels without pulling the lexicon and the municipality catalog.
*/
/** Topic keyword lexicon — terms are compared against normalized text. */
const SPEECH_TOPIC_TERMS: Record<SpeechTopic, readonly string[]> = {
saude: [
'saude',
'sus',
'hospital',
'posto de saude',
'medicamento',
'farmacia popular',
'vacina',
'vacinacao',
'medico',
'enfermagem',
'atencao basica',
'ministerio da saude',
'agentes comunitarios',
],
educacao: [
'educacao',
'escola',
'professor',
'professora',
'estudante',
'aluno',
'universidade',
'ensino',
'alfabetizacao',
'creche',
'merenda',
'pne',
'ifba',
'ufba',
'enem',
],
cultura: [
'cultura',
'cultural',
'artista',
'teatro',
'cinema',
'musica',
'patrimonio',
'lei rouanet',
'carnaval',
'museu',
'livro',
'leitura',
'audiovisual',
],
esporte: ['esporte', 'esportivo', 'futebol', 'atleta', 'olimpiada', 'campeonato'],
'seguranca-publica': [
'seguranca publica',
'seguranca',
'policia',
'policial',
'violencia',
'crime',
'criminalidade',
'trafico',
'armas',
'milicia',
],
'meio-ambiente': [
'meio ambiente',
'ambiental',
'desmatamento',
'garimpo',
'queimada',
'clima',
'climatico',
'sustentabilidade',
'reciclagem',
'poluicao',
'floresta',
'amazonia',
],
'economia-trabalho': [
'economia',
'emprego',
'desemprego',
'trabalhador',
'trabalhadora',
'salario',
'salario minimo',
'renda',
'industria',
'comercio',
'inflacao',
'juros',
'tributacao',
'reforma tributaria',
],
'direitos-humanos': [
'direitos humanos',
'assistencia social',
'bolsa familia',
'cras',
'creas',
'pobreza',
'miseria',
'desigualdade',
'populacao de rua',
'idoso',
'seguridade',
],
infraestrutura: [
'infraestrutura',
'transporte',
'rodovia',
'estrada',
'saneamento',
'mobilidade',
'metro',
'ferrovia',
'porto',
'aeroporto',
'energia eletrica',
'luz para todos',
'pavimentacao',
],
'ciencia-tecnologia': [
'ciencia',
'cientifico',
'tecnologia',
'pesquisa',
'inovacao',
'embrapa',
'internet',
'banda larga',
'inteligencia artificial',
],
'politica-instituicoes': [
'politica',
'politico',
'governo',
'congresso',
'camara dos deputados',
'senado',
'eleicao',
'eleitor',
'democracia',
'instituicoes',
'constituicao',
'impeachment',
'mandato',
'deputado',
'deputada',
'presidente da republica',
'ministro',
'golpe',
'ditadura',
],
agricultura: [
'agricultura',
'agropecuaria',
'agricultor',
'agricultora',
'agricultura familiar',
'pecuaria',
'lavoura',
'safra',
'reforma agraria',
'rural',
'irrigacao',
'seca',
'semiarido',
],
'habitacao-cidades': [
'habitacao',
'moradia',
'minha casa minha vida',
'cidade',
'urbano',
'urbana',
'favela',
'periferia',
'regularizacao fundiaria',
'aluguel',
],
'comunicacao-midia': [
'comunicacao',
'midia',
'imprensa',
'jornal',
'jornalismo',
'radio',
'televisao',
'redes sociais',
'fake news',
'desinformacao',
],
'igualdade-racial': [
'igualdade racial',
'racial',
'racismo',
'racista',
'negro',
'negra',
'quilombola',
'quilombo',
'consciencia negra',
'cotas',
],
'mulheres-genero': [
'mulher',
'mulheres',
'genero',
'feminicidio',
'violencia domestica',
'violencia contra a mulher',
'igualdade de genero',
'maternidade',
'machismo',
],
juventude: [
'juventude',
'jovem',
'jovens',
'adolescente',
'crianca',
'primeira infancia',
'juventude rural',
],
'pessoa-deficiencia': [
'pessoa com deficiencia',
'deficiencia',
'deficiente',
'acessibilidade',
'autismo',
'autista',
'cadeirante',
'pcd',
'libras',
],
}
const SPEECH_SCOPE_TERMS: Record<SpeechScope, readonly string[]> = {
bahia: ['bahia', 'baiano', 'baiana', 'baianos', 'baianas'],
brasil: [
'brasil',
'brasileiro',
'brasileira',
'brasileiros',
'brasileiras',
'nacional',
'pais',
'governo federal',
'uniao',
'republica',
],
internacional: [
'internacional',
'mundo',
'mundial',
'exterior',
'onu',
'mercosul',
'otan',
'uniao europeia',
'china',
'estados unidos',
'eua',
'europa',
'africa',
'america latina',
'guerra',
'refugiados',
'embaixada',
'tratado',
],
}
const escapeRegExp = (value: string): string => value.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')
const containsTerm = (normalizedText: string, term: string): boolean =>
new RegExp(`(^|[^a-z0-9])${escapeRegExp(term)}($|[^a-z0-9])`).test(normalizedText)
// ---------------------------------------------------------------------------
// Municipality mentions
// ---------------------------------------------------------------------------
/**
* NFD-stripped, whitespace-collapsed but CASE-PRESERVING copy, index-aligned
* with `normalizeForSearch` (lowercasing does not change length in pt-BR).
* Used to reject lowercase common words that are also municipality names
* ("saúde" the noun vs. Saúde (BA), "central" the adjective vs. Central).
*/
const stripToComparable = (value: string): string =>
value
.normalize('NFD')
.replace(/[\u0300-\u036f]/g, '')
.replace(/\s+/g, ' ')
.trim()
type MunicipalityMatcher = { normalized: string; entries: MunicipalityCatalogEntry[] }
let municipalityMatchers: readonly MunicipalityMatcher[] | null = null
const getMunicipalityMatchers = (): readonly MunicipalityMatcher[] => {
if (municipalityMatchers) return municipalityMatchers
const byCity = new Map<string, MunicipalityCatalogEntry[]>()
for (const entry of municipalityCatalog) {
const list = byCity.get(entry.city)
if (list) list.push(entry)
else byCity.set(entry.city, [entry])
}
municipalityMatchers = [...byCity.entries()]
.map(([city, entries]) => ({ normalized: normalizeForSearch(city), entries }))
.sort((left, right) => right.normalized.length - left.normalized.length)
return municipalityMatchers
}
/**
* Institutional phrases that contain a municipality name but are not a
* mention of the place (Saúde (BA) inside "Sistema Único de Saúde", Central
* (BA) inside "Banco Central", Planalto (BA) inside "Palácio do Planalto").
* Masked (same length) before matching so the capitalization heuristic does
* not turn them into false positives.
*/
const MUNICIPALITY_FALSE_POSITIVE_PHRASES: readonly string[] = [
'sistema unico de saude',
'ministerio da saude',
'secretaria de saude',
'secretaria municipal de saude',
'conselho de saude',
'conselho municipal de saude',
'saude publica',
'saude bucal',
'atencao basica',
'profissionais de saude',
'plano de saude',
'palacio do planalto',
'planalto central',
'banco central',
'central unica',
'central sindical',
'central de',
]
const maskFalsePositivePhrases = (value: string): string => {
let masked = value
for (const phrase of MUNICIPALITY_FALSE_POSITIVE_PHRASES) {
masked = masked.replace(new RegExp(escapeRegExp(phrase), 'gi'), (match) =>
' '.repeat(match.length),
)
}
return masked
}
/**
* Matches catalog municipality names in the text. Longest names are consumed
* first so "São Félix do Coribe" never also yields "Coribe"; a match must
* start with an uppercase letter in the original text, so prose "saúde" does
* not become the municipality Saúde. Salvador has no city row — a mention of
* the city maps to its 19 zone entries (documented product decision).
*/
export const matchMunicipalityMentions = (text: string): MunicipalityCatalogEntry[] => {
const comparable = maskFalsePositivePhrases(stripToComparable(String(text ?? '')))
if (comparable === '') return []
const masked = comparable.toLowerCase().split('')
const matched: MunicipalityCatalogEntry[] = []
const seen = new Set<string>()
for (const matcher of getMunicipalityMatchers()) {
const pattern = new RegExp(`(^|[^a-z0-9])${escapeRegExp(matcher.normalized)}($|[^a-z0-9])`, 'g')
const haystack = masked.join('')
let found = false
let match: RegExpExecArray | null
while ((match = pattern.exec(haystack)) !== null) {
const start = match.index + match[1].length
if (!/[A-Z]/.test(comparable[start] ?? '')) continue
found = true
for (let index = start; index < start + matcher.normalized.length; index += 1) {
masked[index] = ' '
}
}
if (!found) continue
for (const entry of matcher.entries) {
if (seen.has(entry.slug)) continue
seen.add(entry.slug)
matched.push(entry)
}
}
return matched
}
// ---------------------------------------------------------------------------
// Classification
// ---------------------------------------------------------------------------
export type SpeechFacetInput = {
transcript: string
summary?: string | null
keywords?: readonly string[]
}
/**
* Offline facet pass: topic lexicon + scope lexicon + municipality gazetteer.
* People/programs/projects require the LLM pass; this pass never invents them.
*/
export const classifySpeechByGazetteer = (input: SpeechFacetInput): SpeechFacetClassification => {
const text = [input.summary ?? '', input.keywords?.join(' ') ?? '', input.transcript].join(' ')
const normalized = normalizeForSearch(text)
const topics = SPEECH_TOPICS.filter(({ value }) =>
SPEECH_TOPIC_TERMS[value].some((term) => containsTerm(normalized, term)),
).map(({ value }) => value)
const municipalities = matchMunicipalityMentions(text)
const matchedScopes = new Set<SpeechScope>(
SPEECH_SCOPES.filter(({ value }) =>
SPEECH_SCOPE_TERMS[value].some((term) => containsTerm(normalized, term)),
).map(({ value }) => value),
)
if (municipalities.length > 0) matchedScopes.add('bahia')
return {
topics,
scopes: SPEECH_SCOPES.filter(({ value }) => matchedScopes.has(value)).map(({ value }) => value),
municipalities,
people: [],
programs: [],
projects: [],
}
}

31
src/lib/speechSearch.ts Normal file
View file

@ -0,0 +1,31 @@
/**
* Pure text normalization for the speech catalog (C153). The segment hook
* stores `normalizeForSearch(text)` in `speechSegment.searchText` and the
* search UI (C154) normalizes the query with the SAME function before the
* `contains` query, so accent/case variants match the trigram index.
*/
/** NFD-strips accents, lowercases and collapses whitespace. */
export const normalizeForSearch = (value: string): string =>
value
.normalize('NFD')
.replace(/[\u0300-\u036f]/g, '')
.toLowerCase()
.replace(/\s+/g, ' ')
.trim()
/**
* Dedupes values by their normalized form, preserving the first spelling.
* Used by the facet classifier before persisting people/program mentions.
*/
export const uniqueByNormalizedForm = (values: readonly string[]): string[] => {
const seen = new Set<string>()
const result: string[] = []
for (const value of values) {
const key = normalizeForSearch(value)
if (key === '' || seen.has(key)) continue
seen.add(key)
result.push(value.trim())
}
return result
}

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,158 @@
import { MigrateUpArgs, MigrateDownArgs, sql } from '@payloadcms/db-postgres'
/**
* C153 — speech catalog tables + the `communicator` campaign role.
*
* The generated migration also re-emitted
* `DROP INDEX google_calendar_sync_oauth_connected_at_idx`, a snapshot drift
* from C149 (the index appears in that migration's JSON but its SQL never
* created it, so the statement would fail on every environment); it was
* removed by hand.
*/
export async function up({ db, payload, req }: MigrateUpArgs): Promise<void> {
await db.execute(sql`
CREATE TYPE "public"."enum_speech_topics" AS ENUM('saude', 'educacao', 'cultura', 'esporte', 'seguranca-publica', 'meio-ambiente', 'economia-trabalho', 'direitos-humanos', 'infraestrutura', 'ciencia-tecnologia', 'politica-instituicoes', 'agricultura', 'habitacao-cidades', 'comunicacao-midia', 'igualdade-racial', 'mulheres-genero', 'juventude', 'pessoa-deficiencia');
CREATE TYPE "public"."enum_speech_scopes" AS ENUM('bahia', 'brasil', 'internacional');
CREATE TYPE "public"."enum_speech_legislature" AS ENUM('54', '55', '56', '57');
CREATE TYPE "public"."enum_speech_classified_by" AS ENUM('gazetteer', 'llm', 'manual');
ALTER TYPE "public"."enum_campaign_user_role" ADD VALUE 'communicator' BEFORE 'leader';
CREATE TABLE "speech_topics" (
"order" integer NOT NULL,
"parent_id" integer NOT NULL,
"value" "enum_speech_topics",
"id" serial PRIMARY KEY NOT NULL
);
CREATE TABLE "speech_scopes" (
"order" integer NOT NULL,
"parent_id" integer NOT NULL,
"value" "enum_speech_scopes",
"id" serial PRIMARY KEY NOT NULL
);
CREATE TABLE "speech" (
"id" serial PRIMARY KEY NOT NULL,
"source_key" varchar NOT NULL,
"speech_at" varchar NOT NULL,
"year" numeric,
"legislature" "enum_speech_legislature",
"type" varchar,
"phase" varchar,
"duration_seconds" numeric,
"summary" varchar,
"official_transcript" varchar,
"official_text_url" varchar,
"event_id" numeric,
"event_type" varchar,
"event_start_at" varchar,
"event_end_at" varchar,
"youtube_url" varchar,
"presiding_officer" varchar,
"audio_id" numeric,
"excerpt_t_ms" numeric,
"vod_playback_url" varchar,
"vod_download_url" varchar,
"classified_by" "enum_speech_classified_by" DEFAULT 'gazetteer' NOT NULL,
"updated_at" timestamp(3) with time zone DEFAULT now() NOT NULL,
"created_at" timestamp(3) with time zone DEFAULT now() NOT NULL
);
CREATE TABLE "speech_texts" (
"id" serial PRIMARY KEY NOT NULL,
"order" integer NOT NULL,
"parent_id" integer NOT NULL,
"path" varchar NOT NULL,
"text" varchar
);
CREATE TABLE "speech_rels" (
"id" serial PRIMARY KEY NOT NULL,
"order" integer,
"parent_id" integer NOT NULL,
"path" varchar NOT NULL,
"municipality_id" integer
);
CREATE TABLE "speech_segment" (
"id" serial PRIMARY KEY NOT NULL,
"speech_id" integer NOT NULL,
"order" numeric NOT NULL,
"start_seconds" numeric NOT NULL,
"end_seconds" numeric NOT NULL,
"text" varchar NOT NULL,
"search_text" varchar NOT NULL,
"updated_at" timestamp(3) with time zone DEFAULT now() NOT NULL,
"created_at" timestamp(3) with time zone DEFAULT now() NOT NULL
);
ALTER TABLE "payload_locked_documents_rels" ADD COLUMN "speech_id" integer; ALTER TABLE "payload_locked_documents_rels" ADD COLUMN "speech_segment_id" integer;
ALTER TABLE "speech_topics" ADD CONSTRAINT "speech_topics_parent_fk" FOREIGN KEY ("parent_id") REFERENCES "public"."speech"("id") ON DELETE cascade ON UPDATE no action;
ALTER TABLE "speech_scopes" ADD CONSTRAINT "speech_scopes_parent_fk" FOREIGN KEY ("parent_id") REFERENCES "public"."speech"("id") ON DELETE cascade ON UPDATE no action;
ALTER TABLE "speech_texts" ADD CONSTRAINT "speech_texts_parent_fk" FOREIGN KEY ("parent_id") REFERENCES "public"."speech"("id") ON DELETE cascade ON UPDATE no action;
ALTER TABLE "speech_rels" ADD CONSTRAINT "speech_rels_parent_fk" FOREIGN KEY ("parent_id") REFERENCES "public"."speech"("id") ON DELETE cascade ON UPDATE no action;
ALTER TABLE "speech_rels" ADD CONSTRAINT "speech_rels_municipality_fk" FOREIGN KEY ("municipality_id") REFERENCES "public"."municipality"("id") ON DELETE cascade ON UPDATE no action;
ALTER TABLE "speech_segment" ADD CONSTRAINT "speech_segment_speech_id_speech_id_fk" FOREIGN KEY ("speech_id") REFERENCES "public"."speech"("id") ON DELETE set null ON UPDATE no action;
CREATE INDEX "speech_topics_order_idx" ON "speech_topics" USING btree ("order");
CREATE INDEX "speech_topics_parent_idx" ON "speech_topics" USING btree ("parent_id");
CREATE INDEX "speech_topics_value_idx" ON "speech_topics" USING btree ("value");
CREATE INDEX "speech_scopes_order_idx" ON "speech_scopes" USING btree ("order");
CREATE INDEX "speech_scopes_parent_idx" ON "speech_scopes" USING btree ("parent_id");
CREATE INDEX "speech_scopes_value_idx" ON "speech_scopes" USING btree ("value");
CREATE UNIQUE INDEX "speech_source_key_idx" ON "speech" USING btree ("source_key");
CREATE INDEX "speech_speech_at_idx" ON "speech" USING btree ("speech_at");
CREATE INDEX "speech_year_idx" ON "speech" USING btree ("year");
CREATE INDEX "speech_legislature_idx" ON "speech" USING btree ("legislature");
CREATE INDEX "speech_phase_idx" ON "speech" USING btree ("phase");
CREATE INDEX "speech_event_id_idx" ON "speech" USING btree ("event_id");
CREATE INDEX "speech_audio_id_idx" ON "speech" USING btree ("audio_id");
CREATE INDEX "speech_excerpt_t_ms_idx" ON "speech" USING btree ("excerpt_t_ms");
CREATE INDEX "speech_classified_by_idx" ON "speech" USING btree ("classified_by");
CREATE INDEX "speech_updated_at_idx" ON "speech" USING btree ("updated_at");
CREATE INDEX "speech_created_at_idx" ON "speech" USING btree ("created_at");
CREATE INDEX "speech_texts_order_parent" ON "speech_texts" USING btree ("order","parent_id");
CREATE INDEX "speech_rels_order_idx" ON "speech_rels" USING btree ("order");
CREATE INDEX "speech_rels_parent_idx" ON "speech_rels" USING btree ("parent_id");
CREATE INDEX "speech_rels_path_idx" ON "speech_rels" USING btree ("path");
CREATE INDEX "speech_rels_municipality_id_idx" ON "speech_rels" USING btree ("municipality_id");
CREATE INDEX "speech_segment_speech_idx" ON "speech_segment" USING btree ("speech_id");
CREATE INDEX "speech_segment_updated_at_idx" ON "speech_segment" USING btree ("updated_at");
CREATE INDEX "speech_segment_created_at_idx" ON "speech_segment" USING btree ("created_at");
ALTER TABLE "payload_locked_documents_rels" ADD CONSTRAINT "payload_locked_documents_rels_speech_fk" FOREIGN KEY ("speech_id") REFERENCES "public"."speech"("id") ON DELETE cascade ON UPDATE no action;
ALTER TABLE "payload_locked_documents_rels" ADD CONSTRAINT "payload_locked_documents_rels_speech_segment_fk" FOREIGN KEY ("speech_segment_id") REFERENCES "public"."speech_segment"("id") ON DELETE cascade ON UPDATE no action;
CREATE INDEX "payload_locked_documents_rels_speech_id_idx" ON "payload_locked_documents_rels" USING btree ("speech_id");
CREATE INDEX "payload_locked_documents_rels_speech_segment_id_idx" ON "payload_locked_documents_rels" USING btree ("speech_segment_id");`)
}
export async function down({ db, payload, req }: MigrateDownArgs): Promise<void> {
await db.execute(sql`
ALTER TABLE "speech_topics" DISABLE ROW LEVEL SECURITY;
ALTER TABLE "speech_scopes" DISABLE ROW LEVEL SECURITY;
ALTER TABLE "speech" DISABLE ROW LEVEL SECURITY;
ALTER TABLE "speech_texts" DISABLE ROW LEVEL SECURITY;
ALTER TABLE "speech_rels" DISABLE ROW LEVEL SECURITY;
ALTER TABLE "speech_segment" DISABLE ROW LEVEL SECURITY;
DROP TABLE "speech_topics" CASCADE;
DROP TABLE "speech_scopes" CASCADE;
DROP TABLE "speech" CASCADE;
DROP TABLE "speech_texts" CASCADE;
DROP TABLE "speech_rels" CASCADE;
DROP TABLE "speech_segment" CASCADE;
ALTER TABLE "payload_locked_documents_rels" DROP CONSTRAINT "payload_locked_documents_rels_speech_fk";
ALTER TABLE "payload_locked_documents_rels" DROP CONSTRAINT "payload_locked_documents_rels_speech_segment_fk";
ALTER TABLE "campaign_user" ALTER COLUMN "role" SET DATA TYPE text;
ALTER TABLE "campaign_user" ALTER COLUMN "role" SET DEFAULT 'leader'::text;
DROP TYPE "public"."enum_campaign_user_role";
CREATE TYPE "public"."enum_campaign_user_role" AS ENUM('coordinator', 'advisor', 'candidate', 'leader');
ALTER TABLE "campaign_user" ALTER COLUMN "role" SET DEFAULT 'leader'::"public"."enum_campaign_user_role";
ALTER TABLE "campaign_user" ALTER COLUMN "role" SET DATA TYPE "public"."enum_campaign_user_role" USING "role"::"public"."enum_campaign_user_role";
DROP INDEX "payload_locked_documents_rels_speech_id_idx";
DROP INDEX "payload_locked_documents_rels_speech_segment_id_idx";
ALTER TABLE "payload_locked_documents_rels" DROP COLUMN "speech_id";
ALTER TABLE "payload_locked_documents_rels" DROP COLUMN "speech_segment_id";
DROP TYPE "public"."enum_speech_topics";
DROP TYPE "public"."enum_speech_scopes";
DROP TYPE "public"."enum_speech_legislature";
DROP TYPE "public"."enum_speech_classified_by";`)
}

View file

@ -0,0 +1,21 @@
import { MigrateDownArgs, MigrateUpArgs, sql } from '@payloadcms/db-postgres'
/**
* C153 — the C154 catalog search runs `ILIKE '%term%'` on the normalized
* `speech_segment.search_text`; the GIN trigram index makes it sargable
* instead of a sequential scan per query. Hand-written (not a Payload
* field/schema change) and idempotent — safe to re-run.
*/
export async function up({ db }: MigrateUpArgs): Promise<void> {
await db.execute(sql`CREATE EXTENSION IF NOT EXISTS pg_trgm;`)
await db.execute(
sql`CREATE INDEX IF NOT EXISTS "speech_segment_search_text_trgm_idx" ON "speech_segment" USING gin ("search_text" gin_trgm_ops);`,
)
}
export async function down({ db }: MigrateDownArgs): Promise<void> {
await db.execute(sql`DROP INDEX IF EXISTS "speech_segment_search_text_trgm_idx";`)
// `pg_trgm` is left installed: dropping a cluster-wide extension in a single
// feature's `down` is unnecessarily invasive (same rationale as the contact
// trigram index migration).
}

View file

@ -66,6 +66,8 @@ import * as migration_20260824_000000_reconcile_allocation_decision_rationale_nu
import * as migration_20260824_010000_make_supporter_import_batch_actor_nullable from './20260824_010000_make_supporter_import_batch_actor_nullable';
import * as migration_20260912_052226_add_google_calendar_oauth_connection from './20260912_052226_add_google_calendar_oauth_connection';
import * as migration_20260912_224719_add_activity_public_event from './20260912_224719_add_activity_public_event';
import * as migration_20260913_001112_add_speech_catalog from './20260913_001112_add_speech_catalog';
import * as migration_20260913_001200_add_speech_segment_trgm_index from './20260913_001200_add_speech_segment_trgm_index';
export const migrations = [
{
@ -401,11 +403,21 @@ export const migrations = [
{
up: migration_20260912_052226_add_google_calendar_oauth_connection.up,
down: migration_20260912_052226_add_google_calendar_oauth_connection.down,
name: '20260912_052226_add_google_calendar_oauth_connection'
name: '20260912_052226_add_google_calendar_oauth_connection',
},
{
up: migration_20260912_224719_add_activity_public_event.up,
down: migration_20260912_224719_add_activity_public_event.down,
name: '20260912_224719_add_activity_public_event'
name: '20260912_224719_add_activity_public_event',
},
{
up: migration_20260913_001112_add_speech_catalog.up,
down: migration_20260913_001112_add_speech_catalog.down,
name: '20260913_001112_add_speech_catalog'
},
{
up: migration_20260913_001200_add_speech_segment_trgm_index.up,
down: migration_20260913_001200_add_speech_segment_trgm_index.down,
name: '20260913_001200_add_speech_segment_trgm_index'
},
];

View file

@ -86,6 +86,8 @@ export interface Config {
supporterImportBatch: SupporterImportBatch;
municipalityUpdate: MunicipalityUpdate;
activity: Activity;
speech: Speech;
speechSegment: SpeechSegment;
calendarFeed: CalendarFeed;
googleCalendarSync: GoogleCalendarSync;
electionTally: ElectionTally;
@ -127,6 +129,8 @@ export interface Config {
supporterImportBatch: SupporterImportBatchSelect<false> | SupporterImportBatchSelect<true>;
municipalityUpdate: MunicipalityUpdateSelect<false> | MunicipalityUpdateSelect<true>;
activity: ActivitySelect<false> | ActivitySelect<true>;
speech: SpeechSelect<false> | SpeechSelect<true>;
speechSegment: SpeechSegmentSelect<false> | SpeechSegmentSelect<true>;
calendarFeed: CalendarFeedSelect<false> | CalendarFeedSelect<true>;
googleCalendarSync: GoogleCalendarSyncSelect<false> | GoogleCalendarSyncSelect<true>;
electionTally: ElectionTallySelect<false> | ElectionTallySelect<true>;
@ -272,7 +276,7 @@ export interface CampaignUser {
contact?: (number | null) | Contact;
name: string;
avatar?: (number | null) | Media;
role: 'coordinator' | 'advisor' | 'candidate' | 'leader';
role: 'coordinator' | 'advisor' | 'candidate' | 'communicator' | 'leader';
/**
* O que o assessor enxerga no /campanha.
*/
@ -909,6 +913,105 @@ export interface MunicipalityUpdate {
updatedAt: string;
createdAt: string;
}
/**
* Acervo de falas do deputado na Câmara. Dados e vídeos da Câmara dos Deputados (CC BY 4.0).
*
* This interface was referenced by `Config`'s JSON-Schema
* via the `definition` "speech".
*/
export interface Speech {
id: number;
/**
* Identidade da fala na API da Câmara (data/hora + tipo + fase).
*/
sourceKey: string;
/**
* Horário local de Brasília, como publicado pela Câmara.
*/
speechAt: string;
year?: number | null;
legislature?: ('54' | '55' | '56' | '57') | null;
type?: string | null;
phase?: string | null;
durationSeconds?: number | null;
summary?: string | null;
officialTranscript?: string | null;
officialTextUrl?: string | null;
/**
* Preservadas cruas; as facetas não as substituem.
*/
keywords?: string[] | null;
eventId?: number | null;
eventType?: string | null;
eventStartAt?: string | null;
eventEndAt?: string | null;
youtubeUrl?: string | null;
/**
* Derivado das trocas de mesa da sessão; vazio quando não há registro.
*/
presidingOfficer?: string | null;
audioId?: number | null;
excerptTMs?: number | null;
/**
* Último link conhecido; regerável pelo VOD.
*/
vodPlaybackUrl?: string | null;
/**
* Último link conhecido; regerável pelo VOD.
*/
vodDownloadUrl?: string | null;
topics?:
| (
| 'saude'
| 'educacao'
| 'cultura'
| 'esporte'
| 'seguranca-publica'
| 'meio-ambiente'
| 'economia-trabalho'
| 'direitos-humanos'
| 'infraestrutura'
| 'ciencia-tecnologia'
| 'politica-instituicoes'
| 'agricultura'
| 'habitacao-cidades'
| 'comunicacao-midia'
| 'igualdade-racial'
| 'mulheres-genero'
| 'juventude'
| 'pessoa-deficiencia'
)[]
| null;
scopes?: ('bahia' | 'brasil' | 'internacional')[] | null;
classifiedBy: 'gazetteer' | 'llm' | 'manual';
mentionedMunicipalities?: (number | Municipality)[] | null;
/**
* Menções textuais — não é cadastro de contatos.
*/
mentionedPeople?: string[] | null;
mentionedPrograms?: string[] | null;
mentionedProjects?: string[] | null;
updatedAt: string;
createdAt: string;
}
/**
* This interface was referenced by `Config`'s JSON-Schema
* via the `definition` "speechSegment".
*/
export interface SpeechSegment {
id: number;
speech: number | Speech;
order: number;
startSeconds: number;
endSeconds: number;
text: string;
/**
* Derivado do texto (sem acentos, minúsculas) para a busca por palavra.
*/
searchText: string;
updatedAt: string;
createdAt: string;
}
/**
* This interface was referenced by `Config`'s JSON-Schema
* via the `definition` "calendarFeed".
@ -1446,6 +1549,14 @@ export interface PayloadLockedDocument {
relationTo: 'activity';
value: number | Activity;
} | null)
| ({
relationTo: 'speech';
value: number | Speech;
} | null)
| ({
relationTo: 'speechSegment';
value: number | SpeechSegment;
} | null)
| ({
relationTo: 'calendarFeed';
value: number | CalendarFeed;
@ -1954,6 +2065,56 @@ export interface ActivitySelect<T extends boolean = true> {
updatedAt?: T;
createdAt?: T;
}
/**
* This interface was referenced by `Config`'s JSON-Schema
* via the `definition` "speech_select".
*/
export interface SpeechSelect<T extends boolean = true> {
sourceKey?: T;
speechAt?: T;
year?: T;
legislature?: T;
type?: T;
phase?: T;
durationSeconds?: T;
summary?: T;
officialTranscript?: T;
officialTextUrl?: T;
keywords?: T;
eventId?: T;
eventType?: T;
eventStartAt?: T;
eventEndAt?: T;
youtubeUrl?: T;
presidingOfficer?: T;
audioId?: T;
excerptTMs?: T;
vodPlaybackUrl?: T;
vodDownloadUrl?: T;
topics?: T;
scopes?: T;
classifiedBy?: T;
mentionedMunicipalities?: T;
mentionedPeople?: T;
mentionedPrograms?: T;
mentionedProjects?: T;
updatedAt?: T;
createdAt?: T;
}
/**
* This interface was referenced by `Config`'s JSON-Schema
* via the `definition` "speechSegment_select".
*/
export interface SpeechSegmentSelect<T extends boolean = true> {
speech?: T;
order?: T;
startSeconds?: T;
endSeconds?: T;
text?: T;
searchText?: T;
updatedAt?: T;
createdAt?: T;
}
/**
* This interface was referenced by `Config`'s JSON-Schema
* via the `definition` "calendarFeed_select".
@ -2684,6 +2845,8 @@ export interface TaskCreateCollectionExport {
| 'supporterImportBatch'
| 'municipalityUpdate'
| 'activity'
| 'speech'
| 'speechSegment'
| 'calendarFeed'
| 'googleCalendarSync'
| 'electionTally'

View file

@ -33,6 +33,8 @@ import { Petition } from './collections/Petition'
import { Post } from './collections/Post'
import { PushSubscription } from './collections/PushSubscription'
import { Signature } from './collections/Signature'
import { Speech } from './collections/Speech'
import { SpeechSegment } from './collections/SpeechSegment'
import { StateDeputy } from './collections/StateDeputy'
import { Subscription } from './collections/Subscription'
import { Supporter } from './collections/Supporter'
@ -111,6 +113,8 @@ export default buildConfig({
SupporterImportBatch,
MunicipalityUpdate,
Activity,
Speech,
SpeechSegment,
CalendarFeed,
GoogleCalendarSync,
ElectionTally,

View file

@ -0,0 +1,31 @@
// ---------------------------------------------------------------------------
// Speech catalog (C153) — the "acervo de falas" read surface
// ---------------------------------------------------------------------------
import type { Access } from 'payload'
import { canReadSpeechCatalog } from '@/lib/campaignRoles'
import {
getFreshCampaignUser,
isCampaignUnrestricted,
isPayloadAdmin,
} from '@/utilities/access/shared'
/**
* Readers of the speech catalog: Payload admin or campaign users with the
* `communicator`, `coordinator` or `candidate` role. Advisors and leaders are
* denied — the catalog is not municipality-scoped and not a staff area.
*/
export const canReadSpeech: Access = async ({ req }) => {
if (isPayloadAdmin(req.user)) return true
const currentUser = await getFreshCampaignUser(req)
return currentUser ? canReadSpeechCatalog(currentUser.role) : false
}
/** Manual facet correction — unrestricted roles (and Payload admin) only. */
export const canUpdateSpeech: Access = async ({ req }) => {
if (isPayloadAdmin(req.user)) return true
return isCampaignUnrestricted(await getFreshCampaignUser(req))
}

View file

@ -168,3 +168,5 @@ export {
canSetCalendarFeedSystemField,
canUpdateCalendarFeed,
} from '@/utilities/access/calendarFeeds'
export { canReadSpeech, canUpdateSpeech } from '@/utilities/access/speeches'

View file

@ -10,6 +10,7 @@ export const campaignRoleLabels: Record<CampaignUser['role'], string> = {
coordinator: 'Coordenador Geral',
advisor: 'Assessor',
candidate: 'Candidato',
communicator: 'Assessor de Comunicação',
leader: 'Liderança',
}

View file

@ -0,0 +1,136 @@
import 'server-only'
import { z } from 'zod'
import {
mergeFacetClassification,
parseLlmFacetResponse,
SPEECH_SCOPES,
SPEECH_TOPICS,
type MergedSpeechFacets,
} from '@/lib/speechFacets'
import { classifySpeechByGazetteer, type SpeechFacetInput } from '@/lib/speechGazetteer'
/**
* Facet classification for one speech (C153): the offline gazetteer pass plus
* one LLM refinement on Deep Infra (same key/provider as the ASR). The LLM
* output is validated against the taxonomy; any failure — missing key, HTTP
* error, timeout, invalid JSON — degrades to the gazetteer provenance and
* never throws, so the import continues.
*/
const DEEPINFRA_CHAT_URL = 'https://api.deepinfra.com/v1/openai/chat/completions'
const DEEPINFRA_FACET_MODEL = 'deepseek-ai/DeepSeek-V4-Flash'
const FACET_TIMEOUT_MS = 60_000
const TRANSCRIPT_PROMPT_LIMIT = 12_000
export type SpeechClassificationResult = {
facets: MergedSpeechFacets
llm: {
used: boolean
totalTokens: number | null
estimatedCostUsd: number | null
error: string | null
}
}
const TOPIC_PROMPT_VALUES = SPEECH_TOPICS.map(({ value, label }) => `${value} (${label})`).join(
'; ',
)
const SCOPE_PROMPT_VALUES = SPEECH_SCOPES.map(({ value, label }) => `${value} (${label})`).join(
'; ',
)
const FACET_SYSTEM_PROMPT = [
'Você classifica discursos do deputado federal Jorge Solla (PT-BA) para um catálogo de falas.',
'Responda SOMENTE com um JSON válido, sem markdown, no formato:',
'{"topics": [], "scopes": [], "people": [], "programs": [], "projects": []}',
`topics: subconjunto de [${TOPIC_PROMPT_VALUES}] — use exatamente esses valores.`,
`scopes: subconjunto de [${SCOPE_PROMPT_VALUES}] — use exatamente esses valores.`,
'people: pessoas citadas, nome completo quando houver (máx. 12).',
'programs: programas de governo citados (ex.: "Minha Casa, Minha Vida").',
'projects: proposições citadas (PL, PEC, medida provisória), com o número quando houver.',
'Não invente: liste apenas o que aparece no sumário, nas palavras-chave ou na transcrição.',
'Máximo 4 topics. Sem conteúdo em uma lista, devolva [].',
].join('\n')
const llmEnvelopeSchema = z.object({
choices: z.array(z.object({ message: z.object({ content: z.string() }).passthrough() })).min(1),
usage: z
.object({
total_tokens: z.number().optional(),
estimated_cost: z.number().optional(),
})
.optional(),
})
const buildUserPrompt = (input: SpeechFacetInput): string => {
const transcript = input.transcript.slice(0, TRANSCRIPT_PROMPT_LIMIT)
return [
`SUMÁRIO: ${input.summary ?? '—'}`,
`PALAVRAS-CHAVE OFICIAIS: ${input.keywords?.join('; ') ?? '—'}`,
`TRANSCRIÇÃO (trecho): ${transcript}`,
].join('\n')
}
export const classifySpeech = async (
input: SpeechFacetInput,
): Promise<SpeechClassificationResult> => {
const gazetteer = classifySpeechByGazetteer(input)
const apiKey = process.env.DEEPINFRA_API_KEY
if (!apiKey) {
return {
facets: mergeFacetClassification(gazetteer, null),
llm: { used: false, totalTokens: null, estimatedCostUsd: null, error: 'chave ausente' },
}
}
try {
const response = await fetch(DEEPINFRA_CHAT_URL, {
method: 'POST',
headers: {
Authorization: `Bearer ${apiKey}`,
'Content-Type': 'application/json',
},
body: JSON.stringify({
model: DEEPINFRA_FACET_MODEL,
messages: [
{ role: 'system', content: FACET_SYSTEM_PROMPT },
{ role: 'user', content: buildUserPrompt(input) },
],
response_format: { type: 'json_object' },
temperature: 0,
max_tokens: 500,
}),
signal: AbortSignal.timeout(FACET_TIMEOUT_MS),
})
if (!response.ok) {
throw new Error(`HTTP ${response.status}`)
}
const envelope = llmEnvelopeSchema.safeParse(await response.json())
if (!envelope.success) throw new Error('resposta do provedor fora do formato')
const parsed = parseLlmFacetResponse(JSON.parse(envelope.data.choices[0].message.content))
if (!parsed) throw new Error('resposta fora da taxonomia')
return {
facets: mergeFacetClassification(gazetteer, parsed),
llm: {
used: true,
totalTokens: envelope.data.usage?.total_tokens ?? null,
estimatedCostUsd: envelope.data.usage?.estimated_cost ?? null,
error: null,
},
}
} catch (error) {
return {
facets: mergeFacetClassification(gazetteer, null),
llm: {
used: false,
totalTokens: null,
estimatedCostUsd: null,
error: error instanceof Error ? error.message : String(error),
},
}
}
}

View file

@ -0,0 +1,248 @@
import 'server-only'
import type { Payload, RequiredDataFromCollectionSlug } from 'payload'
import type { MergedSpeechFacets } from '@/lib/speechFacets'
import { normalizeForSearch } from '@/lib/speechSearch'
import type { Speech } from '@/payload-types'
import { withPayloadTransaction } from '@/utilities/payloadTransaction'
/**
* Idempotent persistence of one imported speech (C153). Identity is
* `sourceKey`; metadata is refreshed on every run, segments are replaced only
* when the caller supplies them, and a `manual` facet curation is never
* overwritten. Everything runs in one transaction on the Payload session.
*/
type SpeechSegmentInput = {
startSeconds: number
endSeconds: number
text: string
}
export type SpeechImportBundle = {
sourceKey: string
speechAt: string
year: number | null
legislature: Speech['legislature']
type: string | null
phase: string | null
durationSeconds: number | null
summary: string | null
officialTranscript: string | null
officialTextUrl: string | null
keywords: string[]
eventId: number | null
eventType: string | null
eventStartAt: string | null
eventEndAt: string | null
youtubeUrl: string | null
presidingOfficer: string | null
audioId: number | null
excerptTMs: number | null
vodPlaybackUrl: string | null
vodDownloadUrl: string | null
/** Undefined preserves the stored facets (e.g. a run that skipped classification). */
facets?: MergedSpeechFacets
/** Undefined preserves the stored segments (e.g. `--skip-transcribe`). */
segments?: SpeechSegmentInput[]
}
export type SpeechImportState = {
id: number
audioId: number | null
excerptTMs: number | null
segmentCount: number
classifiedBy: Speech['classifiedBy']
durationSeconds: number | null
vodPlaybackUrl: string | null
vodDownloadUrl: string | null
}
export type SpeechUpsertCounts = {
created: boolean
segmentsInserted: number
segmentsDeleted: number
manualFacetsPreserved: boolean
}
/** Current stored state of one speech, for the importer's skip decisions. */
export const findSpeechImportState = async (
payload: Payload,
sourceKey: string,
): Promise<SpeechImportState | null> => {
const found = await payload.find({
collection: 'speech',
where: { sourceKey: { equals: sourceKey } },
limit: 1,
depth: 0,
select: {
audioId: true,
excerptTMs: true,
classifiedBy: true,
durationSeconds: true,
vodPlaybackUrl: true,
vodDownloadUrl: true,
},
// Intentional bypass: the import CLI is a trusted actor with no session.
overrideAccess: true,
})
const speech = found.docs[0]
if (!speech) return null
const segments = await payload.count({
collection: 'speechSegment',
where: { speech: { equals: speech.id } },
// Intentional bypass: the import CLI is a trusted actor with no session.
overrideAccess: true,
})
return {
id: speech.id,
audioId: speech.audioId ?? null,
excerptTMs: speech.excerptTMs ?? null,
segmentCount: segments.totalDocs,
classifiedBy: speech.classifiedBy,
durationSeconds: speech.durationSeconds ?? null,
vodPlaybackUrl: speech.vodPlaybackUrl ?? null,
vodDownloadUrl: speech.vodDownloadUrl ?? null,
}
}
type SpeechWriteData = RequiredDataFromCollectionSlug<'speech'>
const metadataData = (
bundle: SpeechImportBundle,
): Omit<SpeechWriteData, 'sourceKey' | 'classifiedBy'> => ({
speechAt: bundle.speechAt,
year: bundle.year,
legislature: bundle.legislature,
type: bundle.type,
phase: bundle.phase,
durationSeconds: bundle.durationSeconds,
summary: bundle.summary,
officialTranscript: bundle.officialTranscript,
officialTextUrl: bundle.officialTextUrl,
keywords: bundle.keywords,
eventId: bundle.eventId,
eventType: bundle.eventType,
eventStartAt: bundle.eventStartAt,
eventEndAt: bundle.eventEndAt,
youtubeUrl: bundle.youtubeUrl,
presidingOfficer: bundle.presidingOfficer,
audioId: bundle.audioId,
excerptTMs: bundle.excerptTMs,
vodPlaybackUrl: bundle.vodPlaybackUrl,
vodDownloadUrl: bundle.vodDownloadUrl,
})
export const upsertSpeechBundle = async (
payload: Payload,
bundle: SpeechImportBundle,
): Promise<SpeechUpsertCounts> =>
withPayloadTransaction(payload, async ({ req }) => {
const found = await payload.find({
collection: 'speech',
where: { sourceKey: { equals: bundle.sourceKey } },
limit: 1,
depth: 0,
select: { id: true, classifiedBy: true },
req,
// Intentional bypass: the import CLI is a trusted actor with no session.
overrideAccess: true,
})
const current = found.docs[0]
const manualFacetsPreserved = current?.classifiedBy === 'manual'
const metadata = metadataData(bundle)
const facets: Partial<SpeechWriteData> = {}
if (bundle.facets && !manualFacetsPreserved) {
const municipalitySlugs = bundle.facets.municipalities.map((entry) => entry.slug)
const municipalities =
municipalitySlugs.length === 0
? []
: (
await payload.find({
collection: 'municipality',
where: { slug: { in: municipalitySlugs } },
limit: municipalitySlugs.length,
depth: 0,
select: { slug: true },
req,
// Intentional bypass: resolving gazetteer slugs inside the import transaction.
overrideAccess: true,
})
).docs
facets.topics = bundle.facets.topics
facets.scopes = bundle.facets.scopes
facets.classifiedBy = bundle.facets.classifiedBy
facets.mentionedMunicipalities = municipalities.map((doc) => doc.id)
facets.mentionedPeople = bundle.facets.people
facets.mentionedPrograms = bundle.facets.programs
facets.mentionedProjects = bundle.facets.projects
}
const data = { ...metadata, ...facets }
let speechId: number
let created = false
if (current) {
await payload.update({
collection: 'speech',
id: current.id,
data,
depth: 0,
req,
// Intentional bypass: the import CLI is a trusted actor with no session.
overrideAccess: true,
})
speechId = current.id
} else {
const speech = await payload.create({
collection: 'speech',
data: {
sourceKey: bundle.sourceKey,
classifiedBy: bundle.facets?.classifiedBy ?? 'gazetteer',
...data,
},
depth: 0,
req,
// Intentional bypass: the import CLI is a trusted actor with no session.
overrideAccess: true,
})
speechId = speech.id
created = true
}
let segmentsInserted = 0
let segmentsDeleted = 0
if (bundle.segments !== undefined) {
const removed = await payload.delete({
collection: 'speechSegment',
where: { speech: { equals: speechId } },
req,
// Intentional bypass: replacing segments owned by the speech being imported.
overrideAccess: true,
})
segmentsDeleted = removed.docs.length
for (const [index, segment] of bundle.segments.entries()) {
await payload.create({
collection: 'speechSegment',
data: {
speech: speechId,
order: index + 1,
startSeconds: segment.startSeconds,
endSeconds: segment.endSeconds,
text: segment.text,
searchText: normalizeForSearch(segment.text),
},
req,
// Intentional bypass: the import CLI is a trusted actor with no session.
overrideAccess: true,
})
segmentsInserted += 1
}
}
return { created, segmentsInserted, segmentsDeleted, manualFacetsPreserved }
})

View file

@ -0,0 +1,277 @@
// @vitest-environment node
import { randomUUID } from 'node:crypto'
import type { Payload, RequiredDataFromCollectionSlug } from 'payload'
import { getPayload } from 'payload'
import { afterAll, beforeAll, describe, expect, it } from 'vitest'
import type { Speech } from '@/payload-types'
import config from '@/payload.config'
import { installCampaignFixtures } from '../helpers/campaignFixtures'
let payload: Payload
const createdSpeechIds: number[] = []
const campaignFixtures = installCampaignFixtures({
getPayload: () => payload,
setPayload: (nextPayload) => {
payload = nextPayload
},
})
const createSpeech = async (
overrides: Partial<RequiredDataFromCollectionSlug<'speech'>> = {},
): Promise<Speech> => {
const speech = await payload.create({
collection: 'speech',
data: {
sourceKey: `test-${randomUUID()}`,
speechAt: '2023-02-07T17:28',
type: 'BREVES COMUNICAÇÕES',
phase: 'Breves Comunicações',
summary: 'A defesa do SUS e da saúde pública baiana.',
officialTranscript: 'O SR. JORGE SOLLA (Bloco/PT - BA) - Sr. Presidente...',
keywords: ['Saúde', 'SUS'],
topics: ['saude'],
classifiedBy: 'gazetteer',
...overrides,
},
depth: 0,
overrideAccess: true,
})
createdSpeechIds.push(speech.id)
return speech
}
describe('speech catalog (C153)', () => {
beforeAll(async () => {
payload = await getPayload({ config: await config })
})
afterAll(async () => {
for (const id of createdSpeechIds) {
try {
await payload.delete({ collection: 'speech', id, depth: 0, overrideAccess: true })
} catch {
// already deleted by a cascade assertion
}
}
})
it('derives the year from the wall-clock speechAt', async () => {
const speech = await createSpeech()
expect(speech.year).toBe(2023)
})
it('normalizes the segment text into searchText and matches accent-insensitively', async () => {
const speech = await createSpeech()
const segment = await payload.create({
collection: 'speechSegment',
data: {
speech: speech.id,
order: 1,
startSeconds: 0,
endSeconds: 4.2,
text: 'A defesa do SUS e da saúde pública',
// Deliberately wrong: the beforeValidate hook owns this field.
searchText: 'errado',
},
depth: 0,
overrideAccess: true,
})
expect(segment.searchText).toBe('a defesa do sus e da saude publica')
const hits = await payload.find({
collection: 'speechSegment',
where: { searchText: { contains: 'saude' } },
depth: 0,
overrideAccess: true,
})
expect(hits.docs.map((doc) => doc.id)).toContain(segment.id)
})
it('deletes the speech segments with the speech', async () => {
const speech = await createSpeech()
const segment = await payload.create({
collection: 'speechSegment',
data: {
speech: speech.id,
order: 1,
startSeconds: 0,
endSeconds: 2,
text: 'Trecho qualquer',
searchText: '',
},
depth: 0,
overrideAccess: true,
})
await payload.delete({ collection: 'speech', id: speech.id, depth: 0, overrideAccess: true })
const remaining = await payload.find({
collection: 'speechSegment',
where: { id: { equals: segment.id } },
depth: 0,
overrideAccess: true,
})
expect(remaining.docs).toHaveLength(0)
})
it('reads for communicator, coordinator, candidate and admin; denies advisor, leader and anonymous', async () => {
const speech = await createSpeech()
const fixtures = campaignFixtures()
const communicator = await fixtures.createCampaignUser('communicator')
const coordinator = await fixtures.createCampaignUser('coordinator')
const candidate = await fixtures.createCampaignUser('candidate')
const advisor = await fixtures.createCampaignUser('advisor')
const leader = await fixtures.createCampaignUser('leader')
const admin = await fixtures.createAdminUser()
for (const reader of [communicator, coordinator, candidate, admin]) {
const found = await payload.find({
collection: 'speech',
where: { id: { equals: speech.id } },
depth: 0,
user: reader,
overrideAccess: false,
})
expect(found.docs).toHaveLength(1)
}
for (const denied of [advisor, leader, undefined]) {
await expect(
payload.find({
collection: 'speech',
where: { id: { equals: speech.id } },
depth: 0,
user: denied,
overrideAccess: false,
}),
).rejects.toThrow(/permissão/i)
}
})
it('updates facets for coordinator/candidate/admin but not for communicator/advisor', async () => {
const speech = await createSpeech()
const fixtures = campaignFixtures()
const coordinator = await fixtures.createCampaignUser('coordinator')
const candidate = await fixtures.createCampaignUser('candidate')
const communicator = await fixtures.createCampaignUser('communicator')
const advisor = await fixtures.createCampaignUser('advisor')
const admin = await fixtures.createAdminUser()
for (const editor of [coordinator, candidate, admin]) {
const updated = await payload.update({
collection: 'speech',
id: speech.id,
data: { topics: ['educacao'] },
depth: 0,
user: editor,
overrideAccess: false,
})
expect(updated.topics).toEqual(['educacao'])
}
for (const denied of [communicator, advisor]) {
await expect(
payload.update({
collection: 'speech',
id: speech.id,
data: { topics: ['cultura'] },
depth: 0,
user: denied,
overrideAccess: false,
}),
).rejects.toThrow(/permissão/i)
}
})
it('creates/deletes only for Payload admin', async () => {
const fixtures = campaignFixtures()
const coordinator = await fixtures.createCampaignUser('coordinator')
const admin = await fixtures.createAdminUser()
await expect(
payload.create({
collection: 'speech',
data: {
sourceKey: `test-${randomUUID()}`,
speechAt: '2023-02-08T10:00',
classifiedBy: 'gazetteer',
},
depth: 0,
user: coordinator,
overrideAccess: false,
}),
).rejects.toThrow(/permissão/i)
const speech = await createSpeech()
await expect(
payload.delete({
collection: 'speech',
id: speech.id,
depth: 0,
user: coordinator,
overrideAccess: false,
}),
).rejects.toThrow(/permissão/i)
const adminCreated = await payload.create({
collection: 'speech',
data: {
sourceKey: `test-${randomUUID()}`,
speechAt: '2023-02-08T10:00',
classifiedBy: 'gazetteer',
},
depth: 0,
user: admin,
overrideAccess: false,
})
createdSpeechIds.push(adminCreated.id)
expect(adminCreated.id).toBeGreaterThan(0)
})
it('preserves manual facets on unauthenticated writes and lets admin curation through', async () => {
const speech = await createSpeech({ classifiedBy: 'manual', topics: ['saude'] })
await payload.update({
collection: 'speech',
id: speech.id,
data: { topics: ['cultura'], scopes: ['brasil'], classifiedBy: 'llm' },
depth: 0,
overrideAccess: true,
})
const afterImport = await payload.findByID({
collection: 'speech',
id: speech.id,
depth: 0,
overrideAccess: true,
})
expect(afterImport.topics).toEqual(['saude'])
expect(afterImport.scopes).toEqual([])
expect(afterImport.classifiedBy).toBe('manual')
const admin = await campaignFixtures().createAdminUser()
await payload.update({
collection: 'speech',
id: speech.id,
data: { topics: ['cultura'] },
depth: 0,
user: admin,
overrideAccess: false,
})
const curated = await payload.findByID({
collection: 'speech',
id: speech.id,
depth: 0,
overrideAccess: true,
})
expect(curated.topics).toEqual(['cultura'])
expect(curated.classifiedBy).toBe('manual')
})
})

View file

@ -0,0 +1,206 @@
// @vitest-environment node
import { randomUUID } from 'node:crypto'
import type { Payload } from 'payload'
import { getPayload } from 'payload'
import { afterAll, beforeAll, describe, expect, it } from 'vitest'
import { matchMunicipalityMentions } from '@/lib/speechGazetteer'
import config from '@/payload.config'
import {
findSpeechImportState,
upsertSpeechBundle,
type SpeechImportBundle,
} from '@/utilities/speech/speechImport'
let payload: Payload
const createdSourceKeys = new Set<string>()
const baseBundle = (overrides: Partial<SpeechImportBundle> = {}): SpeechImportBundle => {
const sourceKey = `test-import-${randomUUID()}`
createdSourceKeys.add(sourceKey)
return {
sourceKey,
speechAt: '2023-02-07T17:28',
year: 2023,
legislature: '57',
type: 'BREVES COMUNICAÇÕES',
phase: 'Breves Comunicações',
durationSeconds: 252,
summary: 'Saúde e educação em Feira de Santana.',
officialTranscript: 'O SR. JORGE SOLLA (Bloco/PT - BA) - Sr. Presidente...',
officialTextUrl: null,
keywords: ['SUS'],
eventId: 67091,
eventType: 'Sessão Deliberativa',
eventStartAt: '2023-02-07T15:00',
eventEndAt: '2023-02-07T21:23',
youtubeUrl: null,
presidingOfficer: 'Pompeo de Mattos',
audioId: 558641,
excerptTMs: 1675801808560,
vodPlaybackUrl: null,
vodDownloadUrl: null,
facets: {
topics: ['saude'],
scopes: ['bahia'],
municipalities: matchMunicipalityMentions('Feira de Santana'),
people: ['Lula'],
programs: [],
projects: [],
classifiedBy: 'gazetteer',
},
segments: [{ startSeconds: 0, endSeconds: 2.7, text: 'A saúde pública baiana' }],
...overrides,
}
}
const findBySourceKey = async (sourceKey: string) => {
const found = await payload.find({
collection: 'speech',
where: { sourceKey: { equals: sourceKey } },
depth: 1,
overrideAccess: true,
})
return found.docs[0] ?? null
}
const countSegments = async (speechId: number) => {
const counted = await payload.count({
collection: 'speechSegment',
where: { speech: { equals: speechId } },
overrideAccess: true,
})
return counted.totalDocs
}
describe('upsertSpeechBundle (C153)', () => {
beforeAll(async () => {
payload = await getPayload({ config: await config })
})
afterAll(async () => {
for (const sourceKey of createdSourceKeys) {
const speech = await findBySourceKey(sourceKey)
if (!speech) continue
await payload.delete({ collection: 'speech', id: speech.id, depth: 0, overrideAccess: true })
}
})
it('creates once and replaces segments on re-run without duplicating', async () => {
const bundle = baseBundle()
const first = await upsertSpeechBundle(payload, bundle)
expect(first.created).toBe(true)
expect(first.segmentsInserted).toBe(1)
const speech = await findBySourceKey(bundle.sourceKey)
expect(speech).not.toBeNull()
expect(speech?.topics).toEqual(['saude'])
expect(speech?.mentionedMunicipalities?.[0]).toMatchObject({ slug: 'feira-de-santana' })
const firstSegments = await payload.find({
collection: 'speechSegment',
where: { speech: { equals: speech?.id } },
depth: 0,
overrideAccess: true,
})
expect(firstSegments.docs[0]?.searchText).toBe('a saude publica baiana')
const second = await upsertSpeechBundle(payload, bundle)
expect(second.created).toBe(false)
expect(second.segmentsDeleted).toBe(1)
expect(second.segmentsInserted).toBe(1)
const again = await findBySourceKey(bundle.sourceKey)
expect(again?.id).toBe(speech?.id)
expect(await countSegments(again?.id ?? 0)).toBe(1)
})
it('preserves manual facets and skips facet writes', async () => {
const bundle = baseBundle()
await upsertSpeechBundle(payload, bundle)
const speech = await findBySourceKey(bundle.sourceKey)
await payload.update({
collection: 'speech',
id: speech?.id,
data: { classifiedBy: 'manual', topics: ['cultura'] },
depth: 0,
overrideAccess: true,
})
const counts = await upsertSpeechBundle(
payload,
baseBundle({
sourceKey: bundle.sourceKey,
facets: {
topics: ['saude'],
scopes: ['bahia'],
municipalities: matchMunicipalityMentions('Feira de Santana'),
people: [],
programs: [],
projects: [],
classifiedBy: 'gazetteer',
},
}),
)
expect(counts.manualFacetsPreserved).toBe(true)
const after = await findBySourceKey(bundle.sourceKey)
expect(after?.topics).toEqual(['cultura'])
expect(after?.classifiedBy).toBe('manual')
})
it('preserves the stored facets when the bundle carries none', async () => {
const bundle = baseBundle()
await upsertSpeechBundle(payload, bundle)
const speech = await findBySourceKey(bundle.sourceKey)
await upsertSpeechBundle(
payload,
baseBundle({ sourceKey: bundle.sourceKey, facets: undefined }),
)
const after = await findBySourceKey(bundle.sourceKey)
expect(after?.topics).toEqual(speech?.topics)
expect(after?.classifiedBy).toBe('gazetteer')
})
it('preserves the stored segments when the bundle carries none', async () => {
const bundle = baseBundle()
await upsertSpeechBundle(payload, bundle)
const speech = await findBySourceKey(bundle.sourceKey)
await upsertSpeechBundle(
payload,
baseBundle({ sourceKey: bundle.sourceKey, segments: undefined }),
)
expect(await countSegments(speech?.id ?? 0)).toBe(1)
})
it('reports the stored import state for skip decisions', async () => {
const bundle = baseBundle()
await upsertSpeechBundle(payload, bundle)
const state = await findSpeechImportState(payload, bundle.sourceKey)
expect(state).toMatchObject({
audioId: 558641,
excerptTMs: 1675801808560,
segmentCount: 1,
classifiedBy: 'gazetteer',
})
expect(await findSpeechImportState(payload, 'test-import-missing')).toBeNull()
})
it('keeps a facet-less new record with the default provenance', async () => {
const bundle = baseBundle({ facets: undefined, segments: undefined })
const counts = await upsertSpeechBundle(payload, bundle)
expect(counts.created).toBe(true)
const speech = await findBySourceKey(bundle.sourceKey)
expect(speech?.classifiedBy).toBe('gazetteer')
expect(speech?.topics).toEqual([])
})
})

View file

@ -5,13 +5,19 @@ import { describe, expect, it } from 'vitest'
import {
buildVodUrl,
clockToSeconds,
legislatureForDate,
matchExcerpt,
normalizeSpeakerName,
normalizeTranscription,
parseDurationToSeconds,
parseEventExcerpts,
parseOfficialKeywords,
parsePresidingOfficerTransitions,
parseVodStatus,
resolvePresidingOfficer,
selectSpeechEvents,
speechDate,
speechSourceKey,
speechTimeOfDaySeconds,
} from '../../scripts/lib/camaraSpeeches.mjs'
@ -251,6 +257,146 @@ describe('matchExcerpt', () => {
})
})
describe('speechSourceKey', () => {
it('joins datetime, type and phase into the stable identity', () => {
expect(
speechSourceKey({
dataHoraInicio: '2023-02-07T17:28',
tipoDiscurso: 'BREVES COMUNICAÇÕES',
faseEvento: { titulo: 'Breves Comunicações' },
}),
).toBe('2023-02-07T17:28|BREVES COMUNICAÇÕES|Breves Comunicações')
})
it('degrades missing parts to empty strings', () => {
expect(speechSourceKey({})).toBe('||')
expect(speechSourceKey(null)).toBe('||')
})
})
describe('parseOfficialKeywords', () => {
it('splits the raw newline string and keeps each keyword', () => {
expect(parseOfficialKeywords('Saúde, SUS\r\nEducação\r\n\r\nEscola')).toEqual([
'Saúde, SUS',
'Educação',
'Escola',
])
expect(parseOfficialKeywords(null)).toEqual([])
})
})
describe('legislatureForDate', () => {
it('maps dates to the legislature ranges at the boundaries', () => {
expect(legislatureForDate('2023-02-01')).toBe('57')
expect(legislatureForDate('2023-01-31')).toBe('56')
expect(legislatureForDate('2015-02-01')).toBe('55')
expect(legislatureForDate('2015-01-31')).toBe('54')
expect(legislatureForDate('2011-01-01')).toBe('54')
})
it('returns null outside the supported range or for invalid input', () => {
expect(legislatureForDate('2010-12-31')).toBeNull()
expect(legislatureForDate('')).toBeNull()
expect(legislatureForDate('2023-2-1')).toBeNull()
})
})
describe('parseDurationToSeconds', () => {
it('parses the excerpt card and VOD spellings', () => {
expect(parseDurationToSeconds('0h04\'03"')).toBe(243)
expect(parseDurationToSeconds('1h00\'00"')).toBe(3600)
expect(parseDurationToSeconds('0:04:07')).toBe(247)
expect(parseDurationToSeconds('4:07')).toBe(247)
})
it('rejects empty, malformed and out-of-range values', () => {
expect(parseDurationToSeconds('')).toBeNull()
expect(parseDurationToSeconds(null)).toBeNull()
expect(parseDurationToSeconds('abc')).toBeNull()
expect(parseDurationToSeconds('1::30')).toBeNull()
expect(parseDurationToSeconds('1:99:99')).toBeNull()
expect(parseDurationToSeconds('0h99\'00"')).toBeNull()
expect(parseDurationToSeconds('1:99')).toBeNull()
})
})
describe('parsePresidingOfficerTransitions', () => {
const transitionHtml = (heading: string, times: number[]) => `
<h4 class="g-l-assista__categoria-outros-videos">${heading}</h4>
<ul class="g-l-assista__lista-outros-videos">
${times
.map(
(tMs) =>
`<li><a id="link-trecho-video" href="https://www.camara.leg.br/evento-legislativo/67091?a&#x3D;558641&amp;t&#x3D;${tMs}&trechosOrador=&crawl=no" class="chamada__link-trecho" titulo="X">y</a></li>`,
)
.join('\n')}
</ul>`
it('reads the transition groups, using the newest clip of each as the change moment', () => {
const html = [
transitionHtml(
'Troca da mesa Presidente Bohn Gass por Participante Pompeo de Mattos',
[1675815220717, 1675814951403],
),
transitionHtml(
'Troca da mesa Presidente Pompeo de Mattos por Participante Bohn Gass',
[1675814710153],
),
].join('\n')
expect(parsePresidingOfficerTransitions(html)).toEqual([
{ from: 'Pompeo de Mattos', to: 'Bohn Gass', tMs: 1675814710153 },
{ from: 'Bohn Gass', to: 'Pompeo de Mattos', tMs: 1675815220717 },
])
})
it('returns [] when the page has no transitions or the markup changes', () => {
const html = excerptHtml(67091, {
audioId: 5,
tMs: 1000,
speaker: 'JORGE SOLLA',
party: 'DEPUTADO (PT-BA)',
startTime: '17:30',
duration: '0h01\'00"',
})
expect(parsePresidingOfficerTransitions(html)).toEqual([])
expect(parsePresidingOfficerTransitions('')).toEqual([])
})
})
describe('resolvePresidingOfficer', () => {
// Real transition groups of event 67091 (2023-02-07), newest-clip times.
const transitions = [
{ from: 'Bohn Gass', to: 'Pompeo de Mattos', tMs: 1675815220717 },
{ from: 'Gilberto Nascimento', to: 'Pompeo de Mattos', tMs: 1675800828283 },
{ from: 'Pompeo de Mattos', to: 'Adriana Ventura', tMs: 1675804114707 },
{ from: 'Adriana Ventura', to: 'Pompeo de Mattos', tMs: 1675804616513 },
{ from: 'Pompeo de Mattos', to: 'Bohn Gass', tMs: 1675814710153 },
]
it('returns the incoming officer of the latest transition before the moment', () => {
// Solla spoke at 17:28 BRT (20:28 UTC) — Gilberto→Pompeo at 17:13 BRT.
expect(resolvePresidingOfficer(transitions, Date.parse('2023-02-07T20:28:00Z'))).toBe(
'Pompeo de Mattos',
)
// After Bohn→Pompeo (21:13 BRT), Pompeo presided again.
expect(resolvePresidingOfficer(transitions, Date.parse('2023-02-08T00:16:00Z'))).toBe(
'Pompeo de Mattos',
)
})
it('returns the outgoing officer before the first transition', () => {
expect(resolvePresidingOfficer(transitions, Date.parse('2023-02-07T19:00:00Z'))).toBe(
'Gilberto Nascimento',
)
})
it('returns null without transitions or a usable moment', () => {
expect(resolvePresidingOfficer([], 1675801680000)).toBeNull()
expect(resolvePresidingOfficer(transitions, Number.NaN)).toBeNull()
})
})
describe('normalizeTranscription', () => {
it('accepts the OpenAI-compat segments shape', () => {
expect(

View file

@ -0,0 +1,158 @@
// @vitest-environment node
import { describe, expect, it } from 'vitest'
import { mergeFacetClassification, parseLlmFacetResponse } from '@/lib/speechFacets'
import { classifySpeechByGazetteer, matchMunicipalityMentions } from '@/lib/speechGazetteer'
describe('classifySpeechByGazetteer', () => {
it('detects topics, explicit scopes and municipality mentions', () => {
const result = classifySpeechByGazetteer({
summary: 'Investimentos em saúde e educação na Bahia.',
keywords: ['SUS', 'Escola'],
transcript: 'Defendemos o SUS, a escola pública e a universidade em Feira de Santana.',
})
expect(result.topics).toContain('saude')
expect(result.topics).toContain('educacao')
expect(result.scopes).toContain('bahia')
expect(result.municipalities.map((entry) => entry.slug)).toContain('feira-de-santana')
expect(result.people).toEqual([])
expect(result.programs).toEqual([])
expect(result.projects).toEqual([])
})
it('matches whole words only (no "sus" inside "suspeito")', () => {
const result = classifySpeechByGazetteer({
transcript: 'O suspeito foi ouvido e liberado.',
})
expect(result.topics).not.toContain('saude')
})
it('infers the Bahia scope from a municipality mention alone', () => {
const result = classifySpeechByGazetteer({
transcript: 'A cidade de Vitória da Conquista recebeu o investimento.',
})
expect(result.scopes).toEqual(['bahia'])
})
})
describe('matchMunicipalityMentions', () => {
it('consumes the longest name (São Félix do Coribe never also yields Coribe)', () => {
const slugs = matchMunicipalityMentions('Ele passou por São Félix do Coribe.').map(
(entry) => entry.slug,
)
expect(slugs).toEqual(['sao-felix-do-coribe'])
})
it('does not split a longer name into a shorter one (Condeúba is not Conde)', () => {
const slugs = matchMunicipalityMentions('A comitiva esteve em Condeúba.').map(
(entry) => entry.slug,
)
expect(slugs).toEqual(['condeuba'])
})
it('requires a capitalized proper noun (prose "saúde" is not the municipality Saúde)', () => {
expect(matchMunicipalityMentions('A saúde pública é prioridade.')).toEqual([])
expect(
matchMunicipalityMentions('A agenda passou por Saúde, no interior da Bahia.').map(
(entry) => entry.slug,
),
).toEqual(['saude'])
})
it('ignores institutional phrases that contain a municipality name', () => {
expect(matchMunicipalityMentions('O Sistema Único de Saúde atende a população.')).toEqual([])
expect(matchMunicipalityMentions('O Banco Central elevou os juros.')).toEqual([])
expect(matchMunicipalityMentions('O Palácio do Planalto confirmou a agenda.')).toEqual([])
expect(
matchMunicipalityMentions('Ele esteve em Central, no interior da Bahia.').map(
(entry) => entry.slug,
),
).toEqual(['central'])
})
it('maps a Salvador mention to its 19 zone entries', () => {
const slugs = matchMunicipalityMentions('O comício foi em Salvador.').map((entry) => entry.slug)
expect(slugs).toHaveLength(19)
expect(slugs.every((slug) => slug.startsWith('salvador-ze-'))).toBe(true)
})
})
describe('parseLlmFacetResponse', () => {
it('maps labels and values to the taxonomy, dropping unknown values', () => {
expect(
parseLlmFacetResponse({
topics: ['Saúde', 'seguranca-publica', 'inexistente'],
scopes: ['Bahia', 'Mundo'],
people: ['Dilma Rousseff', 'dilma rousseff', ''],
programs: ['Minha Casa, Minha Vida'],
projects: ['PEC 45'],
}),
).toEqual({
topics: ['saude', 'seguranca-publica'],
scopes: ['bahia'],
people: ['Dilma Rousseff'],
programs: ['Minha Casa, Minha Vida'],
projects: ['PEC 45'],
})
})
it('maps the long labels too', () => {
const parsed = parseLlmFacetResponse({
topics: ['Direitos Humanos e Assistência Social', 'Infraestrutura e Transporte'],
})
expect(parsed?.topics).toEqual(['direitos-humanos', 'infraestrutura'])
})
it('returns null for a response without any known key', () => {
expect(parseLlmFacetResponse(null)).toBeNull()
expect(parseLlmFacetResponse('texto')).toBeNull()
expect(parseLlmFacetResponse({ foo: 1 })).toBeNull()
})
it('caps and truncates the free-text mentions', () => {
const parsed = parseLlmFacetResponse({
people: Array.from({ length: 20 }, (_, index) => `Pessoa ${index}`),
programs: ['x'.repeat(200)],
})
expect(parsed?.people).toHaveLength(12)
expect(parsed?.programs[0]).toHaveLength(120)
})
})
describe('mergeFacetClassification', () => {
const gazetteer = classifySpeechByGazetteer({
transcript: 'O SUS e a escola pública em Feira de Santana.',
})
it('unions topics and prefers the validated LLM scopes', () => {
const merged = mergeFacetClassification(gazetteer, {
topics: ['cultura'],
scopes: ['brasil'],
people: ['Lula'],
programs: [],
projects: [],
})
expect(merged.topics).toEqual([...gazetteer.topics, 'cultura'])
expect(merged.scopes).toEqual(['brasil'])
expect(merged.people).toEqual(['Lula'])
expect(merged.classifiedBy).toBe('llm')
})
it('keeps the gazetteer provenance without an LLM response', () => {
const merged = mergeFacetClassification(gazetteer, null)
expect(merged.classifiedBy).toBe('gazetteer')
expect(merged.scopes).toEqual(gazetteer.scopes)
expect(merged.municipalities).toEqual(gazetteer.municipalities)
})
})

View file

@ -0,0 +1,23 @@
// @vitest-environment node
import { describe, expect, it } from 'vitest'
import { normalizeForSearch, uniqueByNormalizedForm } from '@/lib/speechSearch'
describe('normalizeForSearch', () => {
it('strips accents, lowercases and collapses whitespace', () => {
expect(normalizeForSearch(' Saúde Pública ')).toBe('saude publica')
expect(normalizeForSearch('São Félix do Coribe')).toBe('sao felix do coribe')
expect(normalizeForSearch('')).toBe('')
})
})
describe('uniqueByNormalizedForm', () => {
it('dedupes by normalized form keeping the first spelling', () => {
expect(uniqueByNormalizedForm(['Saúde', 'saude', 'SUS', ' sus '])).toEqual(['Saúde', 'SUS'])
})
it('drops empty values', () => {
expect(uniqueByNormalizedForm(['', ' ', 'Bahia'])).toEqual(['Bahia'])
})
})