mirror of
https://github.com/fsolla/teqo.git
synced 2026-11-12 22:13:05 +00:00
feat(C229): busca por sentido no acervo de falas
Troca o motor do modo tema (Câmara + internet) de expansão lexical para embeddings (Deep Infra, BAAI/bge-m3), com índice próprio (speechEmbedding), ranking por proximidade em memória, evidência real do trecho mais próximo, vazio honesto por limiar e degradação para a busca literal. Inclui a collection oculta + migration, o CLI pnpm acervo:index (--dry-run/--limit/ --source/--force/--probe), o port do design aprovado e a suíte de testes. Pós-rebase: o snapshot da migration seguinte (20260926_062039_add_archive_photo, do C231 já na main) foi regenerado para incluir speech_embedding — sem ele o próximo migrate:create recriaria a tabela. Closes #1360
This commit is contained in:
Francisco Solla
parent
111ab265fe
commit
3e8429daae
49 changed files with 23253 additions and 463 deletions
2
.gitignore
vendored
2
.gitignore
vendored
|
|
@ -84,6 +84,8 @@ next-env.d.ts
|
|||
/data/falas-web/
|
||||
# C231 — Flickr photo-archive cache + receipts (pnpm flickr:import)
|
||||
/data/flickr/
|
||||
# C229 — speech semantic index reports (pnpm acervo:index)
|
||||
/data/acervo-index/
|
||||
# Solla × Ceuci report HTML cache (scripts/build-solla-ceuci-salvador-report.mjs)
|
||||
/data/solla-ceuci-salvador/
|
||||
|
||||
|
|
|
|||
1
docs/changelog/2026-09-26-c229.md
Normal file
1
docs/changelog/2026-09-26-c229.md
Normal file
|
|
@ -0,0 +1 @@
|
|||
**C229 — Busca por sentido no acervo de falas (2026-09-26):** o modo "Por tema" do acervo interno (Câmara + internet) deixou de ser expansão lexical e passou a ser **similaridade por sentido** (embeddings), com o resultado ordenado por proximidade e a evidência real de cada fala. Motor: `src/lib/speechSemantic.ts` (puro — cosine, pooling, janelas de texto, hash de conteúdo, ranking com limiar `SPEECH_SEMANTIC_MIN_COSINE=0.4`, ordenação com relevância default e data/duração do usuário sobrepondo) e a fronteira de provider `deepInfraEmbed.ts` (endpoint OpenAI-compatible da Deep Infra, modelo `BAAI/bge-m3`, mesma `DEEPINFRA_API_KEY` do ASR, batches de 32, timeout, L2-normalização e contrato "nunca lança → `null`"; sem dependência nova). Índice: collection oculta nova `speechEmbedding` (`kind: speech|segment|window`, `speech`, `order`, `startSeconds`, `model`, `dimensions`, `contentHash`, `vector` json; `read: canReadSpeech`, escrita admin-only, `admin.hidden`; cascade no `Speech.beforeDelete`, que também corrige o rollback SQL do C155) com migration gerada `20260926_034118_add_speech_embedding` — o diff gerado trazia drift das migrations de 2026-09-24 cujos snapshots perderam objetos do C215/C219, então o SQL foi hand-reduzido ao escopo e o snapshot novo repara a cadeia para as próximas migrations. Loader: `mode=tema` embute a query uma vez, rankeia em memória os candidatos das **facetas** (sem `LIKE` de `q` e sem C174), fatia a página de 25 e carrega a evidência do melhor trecho (`speechSemanticSearch.ts`); degradação honesta para a busca literal de hoje + aviso discreto quando o provider cai ou não há candidato indexado (anti-falso-positivo), vazio honesto abaixo do limiar, e o ator fora do gate nunca dispara o provider (fail-closed). As duas loaders compartilham um único `loadSemanticSpeechPage`; o provider só é consultado quando o modelo bate (vetor de outro modelo/1024 dims nunca é comparado). CLI: `pnpm acervo:index` (`scripts/index-speeches.mjs` + `scripts/lib/speechIndex.mjs`) idempotente por hash+modelo, com `--dry-run`, `--limit`, `--source`, `--force`, relatório (indexadas/puladas/falhas/trechos/bytes/tokens/custo) e o `--probe "<tema>"` read-only (top-N com score, só CLI) que calibra o limiar e verifica o aceite pós-deploy; runbook §C155/§C225 ganhou o passo de indexação. UI (Impeccable B, port do design `docs/plans/acervo-busca-por-sentido-ui-design.html`): selo "Tema" + "Termo exato" coexistindo, bloco "Trecho mais próximo do tema" (trecho real, sem `<mark>` e sem score), heading/subtítulo responsivos, seletor de ordenação da internet com "Mais relevantes" default no tema (e vocabulário literal no degradado), vazio honesto da cena 03 e selo literal no card degradado; a Câmara mantém a decisão de produto registrada de não ganhar controle de ordenação novo. Testes: unit do motor/planejamento/CLI/provider/URL/VM/cards, int do ranking/limiar/degradação/facetas/modelo/evidência/cascade/indexação ponta-a-ponta e e2e do degradado + ordenação do tema; `pnpm gate:fast`, `knip`, `format:check` e os 41 e2e do `campaignSpeechAcervo` verdes. Crítica do `designer` (trigger c) contra o app renderizado em três ciclos (390/1280, tema/vazio/degradado/exato): todas as cenas/estados certificados; a única divergência material — o menu **aberto** do "Ordenar por" é o `<select>` nativo aprovado no C216 e não o listbox desenhado no artefato — foi levada ao humano no gate e **aceita explicitamente** (conteúdo/ordem/default conferidos por DOM). `Design tier: openai/gpt-5.6-sol`. O aceite de produto ("combate à oposição", "impeachment") depende do corpus real: Fase 5 do impl plan no homeserver (`acervo:index` + `--probe`), com top-N/limiar/custo a registrar aqui na primeira execução.
|
||||
|
|
@ -757,15 +757,46 @@ changelog `docs/changelog/2026-09-13-c155.md`.
|
|||
O import é aditivo e idempotente; não há migração para desfazer. Para remover
|
||||
uma legislatura (ex.: reimportar do zero), apagar os discursos e seus
|
||||
segmentos — as rels (`speech_rels`, `speech_topics`, `speech_scopes`) caem por
|
||||
cascade:
|
||||
cascade (o índice de sentido sai antes; a FK dos vetores é `SET NULL` sobre
|
||||
coluna NOT NULL, então o SQL cru precisa apagá-los por conta própria):
|
||||
|
||||
```sql
|
||||
DELETE FROM "speech_embedding" WHERE "speech_id" IN (SELECT id FROM "speech" WHERE "legislature" = '55');
|
||||
DELETE FROM "speech_segment" WHERE "speech_id" IN (SELECT id FROM "speech" WHERE "legislature" = '55');
|
||||
DELETE FROM "speech" WHERE "legislature" = '55';
|
||||
```
|
||||
|
||||
Para desfazer o acervo inteiro: `DELETE FROM "speech_segment"; DELETE FROM "speech";`
|
||||
(a cobertura `--coverage` volta a zero). Reexecutar `--all` reconstrói.
|
||||
Para desfazer o acervo inteiro: `DELETE FROM "speech_embedding"; DELETE FROM
|
||||
"speech_segment"; DELETE FROM "speech";` (a cobertura `--coverage` volta a
|
||||
zero). Reexecutar `--all` reconstrói, e `pnpm acervo:index` refaz o índice.
|
||||
|
||||
### C229 — índice de sentido do acervo
|
||||
|
||||
O modo **"Por tema"** do acervo (Câmara + internet) passou a ser similaridade
|
||||
por embeddings (C229): as falas importadas precisam entrar no índice vetorial
|
||||
antes de aparecerem nessa busca. O passo é manual, idempotente por hash de
|
||||
conteúdo + modelo e deve ser rodado **depois** de qualquer import (C155/C225):
|
||||
|
||||
```bash
|
||||
# no homeserver, com o env de produção carregado (DEEPINFRA_API_KEY)
|
||||
cd ~/teqo-backfill && git fetch origin && git checkout <SHA> && pnpm install
|
||||
set -a; source ~/stack/.env; set +a
|
||||
set -a; source ~/stack/teqo-1313.env; set +a
|
||||
export DATABASE_URL="${DATABASE_URL/@postgres:5432/@127.0.0.1:5433}"
|
||||
|
||||
# ensaio (read-only): quanto falta indexar e quantos trechos
|
||||
pnpm acervo:index --dry-run
|
||||
# indexação (write guard do C229)
|
||||
ACERVO_INDEX_CONFIRM=1 pnpm acervo:index --source all
|
||||
# calibração/verificação do aceite (read-only; imprime top-N com score)
|
||||
pnpm acervo:index --probe "combate à oposição" --top 10
|
||||
pnpm acervo:index --probe "impeachment" --top 10
|
||||
```
|
||||
|
||||
O relatório JSON fica em `data/acervo-index/reports/` (gitignored) com
|
||||
indexadas/puladas/falhas/trechos/tokens/custo. Os temas do aceite e 3–4 temas
|
||||
de controle validam `SPEECH_SEMANTIC_MIN_COSINE` (`src/lib/speechSemantic.ts`);
|
||||
registre os top-N e o valor final no changelog. Reexecutar só paga o que mudou.
|
||||
|
||||
## C225 — rodada de falas da internet no acervo de produção
|
||||
|
||||
|
|
@ -1050,6 +1081,12 @@ flock -n 9 || { printf '%s\n' 'já existe uma rodada web' >&2; exit 1; }
|
|||
`pending`/`review` que ainda o referencie. Se essa prova não for feita, a
|
||||
rodada fica bloqueada para aprovação operacional.
|
||||
|
||||
7. **Índice de sentido (C229):** depois do import, rode o passo do C229
|
||||
(`ACERVO_INDEX_CONFIRM=1 pnpm acervo:index --source web`, ou `--source all`
|
||||
para pegar as duas origens). Sem ele, as falas novas não aparecem na busca
|
||||
"Por tema"; o `--probe` confere o aceite e o relatório traz tokens/custo.
|
||||
O passo está detalhado na seção "C229 — índice de sentido do acervo".
|
||||
|
||||
### Inventário read-only
|
||||
|
||||
As consultas abaixo não escrevem no banco:
|
||||
|
|
|
|||
177
docs/plans/acervo-busca-por-sentido-impl.md
Normal file
177
docs/plans/acervo-busca-por-sentido-impl.md
Normal file
|
|
@ -0,0 +1,177 @@
|
|||
# Impl: C229 — Busca por sentido no acervo de falas
|
||||
|
||||
Status: aprovado
|
||||
Atualizado em: 2026-09-25
|
||||
Issue: #1360
|
||||
Intenção: docs/plans/acervo-busca-por-sentido.md
|
||||
Appetite restante: herdado (~2–3 dias eng)
|
||||
|
||||
## Leitura da intenção
|
||||
|
||||
- **Outcome:** em `mode=tema`, a busca do acervo de falas (Câmara + internet) passa a ser similaridade por sentido (embeddings), não expansão lexical: lista ordenada por proximidade (data só desempate), cada resultado com o trecho real mais próximo do tema (sem score, sem destaque de termo, sem justificativa gerada); acima do limiar vazio honesto; motor fora → busca literal de hoje com o aviso discreto.
|
||||
- **O que NÃO negociar:** o gate `canReadSpeech`/`canReadCommunicationCatalog` não alarga nem ganha atalho; toda leitura local com `user` + `overrideAccess: false`; a fala continua a unidade (abrir/assistir/baixar intocado); "Termo exato" continua o default e não regride; nenhum score numérico na UI; gravações enviadas (C199/C219), Central pública (S28) e Sollinha (C158) ficam fora; `expandSpeechSearchTheme` não pode ser deletado (gravações/Central/conteúdo o consomem); `openai/*` intocado (OPS123); Consent/LGPD intocados (o único dado externo é o texto da busca e o texto das falas — sem PII de cidadão).
|
||||
- **O que reavaliar:** (a) a hipótese de que a expansão sai do caminho principal está certa, mas o mecanismo de degradação **não** mantém a expansão — degrada para a busca literal pura (sem LLM); (b) o índice não cabe no `speech`/`speechSegment` sem inchar o dono e sem um rebuild limpo — vira collection oculta própria; (c) a evidência semântica exige vetores por trecho (não só por fala): o card lê o melhor trecho real da página; (d) o contrato de `sort` hoje é "web-only e `recentes` nunca serializado" e precisa expressar "Mais relevantes" (default do tema) mantendo os bytes do modo exato.
|
||||
|
||||
## Abordagem recomendada
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
A[modo tema com q] --> B{ator le o catalogo?}
|
||||
B -- nao --> F[payload.find fail-closed]
|
||||
B -- sim --> C[embed da query<br/>DeepInfra bge-m3]
|
||||
C -- null --> DEG[busca literal de hoje<br/>themeUnavailable true]
|
||||
C -- vetor --> D[find de candidatos<br/>facetas e origem, sem perna textual]
|
||||
D --> E[find dos vetores de fala<br/>speechEmbedding kind=speech]
|
||||
E -- zero indexados --> DEG
|
||||
E -- vetores --> G[rank em memoria<br/>cosine desc, data como desempate]
|
||||
G --> H[limiar global + ordenacao escolhida]
|
||||
H --> I[fatia da pagina de 25]
|
||||
I --> J[docs da pagina + melhor trecho por fala]
|
||||
J --> K[VM semanticMatch + trecho real sem destaque]
|
||||
DEG --> L[VM lexical de hoje + aviso discreto]
|
||||
```
|
||||
|
||||
**Opções consideradas:** A) embeddings em memória (DeepInfra, índice derivado em collection oculta, ranking no app) | B) expansão lexical/LLM como motor com reforço de recall | C) pgvector/serviço vetorial dedicado com ANN.
|
||||
**Recomendação:** A — é o pedido literal (similaridade por sentido), reusa a infra de IA já contratada (`DEEPINFRA_API_KEY`), mantém access/`where`/paginação por fala e não exige imagem de Postgres com `vector` nem serviço novo; o índice é derivado, reconstruível e alimentado por CLI manual, como os imports.
|
||||
**Rejeitadas:** B porque é exatamente o comportamento que o dono rejeitou ("parece busca por termo somada a matches em temas pré-definidos") e a intenção manda a expansão no máximo degradar; C porque pgvector não existe na imagem (`postgres:17-alpine`; troca de imagem é mudança de infra/produção fora do appetite) e um serviço vetorial dedicado reabre o rabbit hole nomeado.
|
||||
|
||||
### Componentes / mudanças
|
||||
|
||||
- **`src/utilities/ai/deepInfraEmbed.ts` (novo, `server-only`)** — fronteira do provider: `DEEPINFRA_EMBED_URL = 'https://api.deepinfra.com/v1/openai/embeddings'`, `DEEPINFRA_EMBED_MODEL = 'BAAI/bge-m3'`, `DEEPINFRA_EMBED_DIMENSIONS = 1024`, `DEEPINFRA_EMBED_COST_PER_MILLION_TOKENS_USD` (constante única do relatório), `EMBED_TEXT_MAX_CHARS` (2000), `EMBED_BATCH_SIZE` (32), `embedSpeechTexts(texts)` (input em lote, `encoding_format: 'float'`, parsing por `index`, L2-normaliza, `AbortSignal.timeout(30_000)`, nunca lança → `null`) e `embedSpeechQuery(text)` (1 input, timeout curto de 8s para não segurar a página). Tipo `SpeechQueryEmbeddingResolver`. Segue o padrão `fetch` + timeout + nunca lança de `deepInfraTranscribe.ts`; sem `@ai-sdk/*` nem dependência nova.
|
||||
- **`src/lib/speechSemantic.ts` (novo, puro/client-safe)** — motor: `dotProduct`/`cosineSimilarity` (vetores normalizados na escrita), `normalizeVector`, `poolSpeechVector` (média L2-normalizada dos trechos = vetor da fala), `speechTextWindows` (janelas de 1200 chars com 200 de sobreposição, para fala sem segmento), `speechEmbeddingHash` (FNV-1a incluindo modelo = idempotência do índice), `rankSemanticHits`, `selectSemanticHits` (limiar), `sortSemanticHits` (relevância/recentes/duração), `pickBestSemanticChunk` e `SPEECH_SEMANTIC_MIN_COSINE = 0.4` (valor inicial — D6).
|
||||
- **`src/utilities/speech/speechSemanticSearch.ts` (novo, `server-only`)** — orquestração do caminho semântico: candidatos = `payload.find('speech', { where: buildSpeechFacetWhere(state), select: { id, speechAt, durationSeconds }, pagination: false, user, overrideAccess: false })`; vetores = `payload.find('speechEmbedding', { where: kind=speech e speech in candidatos, user, overrideAccess: false })`; cosine em memória, limiar, ordenação escolhida, slice de `speechPageSize`; busca dos vetores de trecho **da página** e resolução do melhor trecho (segmento por `order`/`startSeconds`, janela recomputada do texto original quando `kind=window`). Devolve `pageHits`, `totalHits`, `indexedCandidates` e `evidenceBySpeech`.
|
||||
- **`src/utilities/speech/speechPageData.ts`** — o 4º parâmetro `expandTheme` vira `embedQuery: SpeechQueryEmbeddingResolver = embedSpeechQuery` nas duas loaders (mesmo padrão de injeção de `loadSpeechDetailPageData`); `resolveSpeechThemeTerms` é removido (expansão sai do acervo); com tema+`q`: gate de papel → `embedQuery` → caminho semântico → docs/segmentos/cortes/facetas da página + evidência; `themeUnavailable` quando o embed da query for `null` **ou** `indexedCandidates === 0`; `themeApplied` passa a significar "o motor de sentido rodou nesta request" (não mais "termos expandidos"); C174 (origem de corte) só é consultado no caminho lexical.
|
||||
- **`src/utilities/speech/speechListFilters.ts`** — remove `themeTerms` de `buildSpeechTextBranches`/`buildSpeechListWhere*` (a perna textual volta a ser só `q`, como no C180) e exporta `buildSpeechFacetWhere` como fronteira única dos candidatos semânticos (sem twinar o where).
|
||||
- **`src/lib/speechSearch.ts`** — só comentário do espelho C180 atualizado (a perna textual volta a ter `q`); predicados intocados.
|
||||
- **`src/utilities/speech/speechListUrl.ts` + `src/lib/acervoListSort.ts`** — contrato de `sort` ciente do modo (D5): `relevancia` reconhecido/omitido só no tema; `recentes` serializado só no tema; demais bytes do modo exato intactos. `ACERVO_THEME_SORT_OPTIONS` (Mais relevantes + os três) e o mapeamento puro "valor do select → estado" vivem no vocabulário compartilhado, mas só o caminho de falas os usa — gravações intocadas.
|
||||
- **`src/utilities/speech/speechViewModels.ts`** — `SpeechListItemViewModel` e `WebSpeechListItemViewModel` ganham `semanticMatch: boolean` e `matchedTextSearch` (o web também), no lugar de `themeMatchTerm` (removido) e de `pickThemeMatch`; no caminho semântico o `excerpt` é o melhor trecho real **sem** destaque (`buildHighlightedExcerpt(source, '')`), `matchKind='theme'` e `watchHref` mira o `t=` do trecho de evidência quando existir.
|
||||
- **UI (`src/components/campaign/speech/` + rota)** — `SpeechResultCard.tsx`: `isTheme = speech.semanticMatch`, bloco renomeado para **"Trecho mais próximo do tema"** (sem `<mark>`), selo "Tema" + "Termo exato" coexistindo; `WebSpeechResultCard.tsx` ganha os mesmos selos e o bloco de evidência preservando título/plataforma/capa (a cena 01 é a fonte internet); `SpeechThemeFallbackNotice.tsx` com a copy do design; na degradação os cards literais mostram o selo "Termo exato" (cena 04) sem alterar a lista do modo exato; `page.tsx` com heading/subtítulo do design ("Resultados por tema" / "Do mais próximo ao menos próximo do sentido buscado."), vazio honesto da cena 03 ("Nenhuma fala combina bem com esse tema" + Reformular/Usar termo exato/Limpar filtros), aviso/retry e `AcervoSortSelect` com as opções de tema na internet.
|
||||
- **Migration:** `add_speech_embedding` (Fase 2, **STOP-POINT**): cria a collection oculta `speechEmbedding` e o índice da FK; commit dos três arquivos (`ts`/`json`/`index.ts`).
|
||||
- **Access / Consent:** nenhuma chave `Consent` nova, nenhum PII; `speechEmbedding.access.read = canReadSpeech`, `create/update/delete = payloadAdminOnly`; o CLI usa `overrideAccess: true` justificado (ator confiável, como os imports); o gate de papel antecede o embed (ator negado nunca dispara chamada externa) e o `find` continua a barreira final.
|
||||
- **CLI:** `scripts/index-speeches.mjs` + puro `scripts/lib/speechIndex.mjs` + `pnpm acervo:index` (`--dry-run`, `--limit`, `--source camara|web|all`, `--probe "<tema>" [--top N]`), relatório JSON em `data/acervo-index/reports/` (gitignore) com indexadas/puladas/falhas/cobertura/tokens/custo; guardas `assertLocalDatabase` + `assertWriteConfirm({ flag: 'ACERVO_INDEX_CONFIRM' })`; o `--probe` é read-only e usa o mesmo ranking do app.
|
||||
- **Manifesto/ops:** entry de `src/collections/SpeechEmbedding.ts` na vertical do acervo em `scripts/lib/e2e-affected-manifest.mjs`; `playwright.config.ts` passa a anular `DEEPINFRA_API_KEY`; `docs/ops/teqo-1313-deploy.md` §C155/§C225 ganham o passo de indexação; uma entrada em `docs/changelog/`.
|
||||
|
||||
### Dados → forma (N/A justificado)
|
||||
|
||||
N/A. O item não apresenta métrica nem agregado: a saída é uma lista de falas com um trecho real; a intenção proíbe score numérico e a forma aprovada no design é "selo + trecho", sem barra/percentual/justificativa gerada.
|
||||
|
||||
## Fases verificáveis
|
||||
|
||||
> **STOP-POINT obrigatório (Fase 2):** schema só entra com aprovação humana da migration. A execução **para** depois de revisar o SQL gerado; só então `pnpm migrate` local, `pnpm generate:types` e a sequência. Nada de Fase 3+ antes disso.
|
||||
|
||||
1. **Fase 1 — tracer offline (sem migration, ~0,5–1 dia):** `lib/speechSemantic.ts`, `utilities/ai/deepInfraEmbed.ts`, contrato de `sort` do tema, VM/cards (`semanticMatch`, "Trecho mais próximo do tema", selos coexistentes) e copy do degradado — tudo com o loader ainda lexical. Testes: `speechSemantic.unit.spec.ts` (cosine, pooling, hash, janelas, limiar, ordenação, melhor trecho), `deepInfraEmbed.unit.spec.ts` (sem chave → `null` e sem fetch; lote por `index`; dimensão errada/HTTP/timeout → `null`; normalização; modelo e `encoding_format` no corpo), `speechListUrl.unit.spec.ts` (relevância canônica no tema; `recentes` serializado só no tema; bytes do modo exato idênticos; `relevancia` fora do tema descartado), `acervoListSort.unit.spec.ts` (opções/mapeamento do tema; gravações intactas), `speechViewModels.unit.spec.ts` e `speechResultCards.unit.spec.tsx` (trecho sem destaque, selos, `watchHref` com `t=`), `speechListFilters.unit.spec.ts` (builder volta ao formato pré-C192).
|
||||
2. **Fase 2 — schema (STOP-POINT, ~0,5 dia):** `src/collections/SpeechEmbedding.ts` (slug `speechEmbedding`, `kind: speech|segment|window`, `speech`, `order`, `startSeconds`, `model`, `dimensions`, `contentHash`, `vector` json, `admin.hidden`), registro no `payload.config.ts`, cascade em `Speech.beforeDelete` (embeddings antes da mídia, `req` + bypass justificado), `pnpm migrate:create add_speech_embedding`. **Revisar o SQL; parar para aprovação humana.** Depois: `pnpm migrate`, `pnpm generate:types`, `pnpm generate:importmap` (se necessário) e int de cascade (apagar fala apaga embeddings e segmentos).
|
||||
3. **Fase 3 — loader + indexação (~0,5–1 dia):** `speechSemanticSearch.ts`, loaders das duas fontes com `embedQuery` injetável, CLI + `--probe`, `.gitignore`. Testes int (sem corpus, com embeddings semeados): (a) fala acima do limiar aparece ordenada por proximidade com `semanticMatch` e evidência real; (b) nada acima do limiar → `rows=[]`, `themeApplied=true`, `themeUnavailable=false` (vazio honesto); (c) `embedQuery → null` → literais + `themeUnavailable=true` (aviso); (d) vetor ok e zero indexados → degrada igual; (e) facetas restringem candidatos; (f) `sort=recentes` no tema ordena por data após o limiar e duração filtra/ordena; (g) `order` fora dos limites → fallback para o primeiro segmento; (h) advisor/leader rejeitados sem chamar o embedder; unit da CLI (`speechIndex.unit.spec.ts`: args, plano, hash/skip, relatório).
|
||||
4. **Fase 4 — UI/gates (~0,5 dia):** port classe-a-classe das cenas 01–04 (heading, sort com "Mais relevantes", vazio, degradado, selo no card degradado), extensão de `tests/e2e/campaignSpeechAcervo.e2e.spec.ts` (degradado com copy nova e sem selo "Tema" fabricado; `source=internet&mode=tema` com "Mais relevantes" e `sort=recentes` canônico sem loop; modo exato inalterado) e entry no manifesto. `pnpm gate:fast`; PR via `pnpm push` (high-risk por `src/migrations` + `src/collections` → curado, que já inclui `campaignSpeechAcervo`).
|
||||
5. **Fase 5 — pós-deploy (aceite de produto, corpus real):** após o deploy de produção, no homeserver, rodar `pnpm acervo:index` (com `ACERVO_INDEX_CONFIRM=1`) e depois `--probe "combate à oposição"` e `--probe "impeachment"`, além de 3–4 temas de controle fora do domínio; registrar top-N/scores/cobertura/custo na Issue e no changelog; ajustar `SPEECH_SEMANTIC_MIN_COSINE` com a evidência se necessário e repetir.
|
||||
|
||||
## Rabbit holes / Não escopo (engenharia)
|
||||
|
||||
- pgvector, imagem de Postgres com `vector`, serviço/banco vetorial dedicado e ANN — rejeitado (infra fora do appetite; sem corpus local).
|
||||
- Reindexação em tempo real/fila/job — imports seguem manuais; a atualização do índice é passo do runbook.
|
||||
- Expansão lexical/LLM como motor ou como fallback — a degradação é a busca literal pura de hoje.
|
||||
- Gravações (C199/C219), Central pública (S28), Sollinha (C158), busca global e dossiês — fora do recorte; `expandSpeechSearchTheme` preservado para eles.
|
||||
- Score/justificativa gerada/por que apareceu numérico — a evidência é trecho real.
|
||||
- Curadoria de sinônimos, cache de embeddings de query, rerank de segundo passe e quantização — deferidos com gatilho (D2/D6).
|
||||
- UI nova de ordenação na Câmara — o modo tema da Câmara é sempre relevância, sem controle de sort.
|
||||
|
||||
## Riscos e mitigação
|
||||
|
||||
- **Limiar mal calibrado (falso-vazio × falso-cheio):** limiar global de 0.40 (D6) calibrado com `--probe` sobre os dois exemplos do aceite e controles; constante única; o vazio é honesto; nenhum resultado abaixo do limiar entra.
|
||||
- **Latência/memória por request** (carrega vetores de todos os candidatos; ~1000 falas no corpus atual): medir no probe; gatilho — p95 > ~1,5s ou memória do container doer → base64 Float32 (4× menor) e/ou quantização int8 (D2), reavaliando pgvector/artefato materializado.
|
||||
- **Índice desatualizado/parcial** (fala importada sem indexar some do tema em silêncio): o CLI é idempotente por hash+modelo, `--dry-run` imprime cobertura/pendências e o runbook encadeia `pnpm acervo:index` após os imports; gatilho — se a defasagem doer, o import chama o indexador (mesma função) em vez de duplicar lógica.
|
||||
- **Evidência desalinhada após reimport** (segmentos trocados sem reindexar): `order` fora dos limites cai no primeiro segmento; o reindex é o conserto; a cobertura do relatório denuncia.
|
||||
- **Provider/chave ausente em produção** (lição do C192): a ausência degrada com o aviso honesto; o deploy já audita `DEEPINFRA_API_KEY` e o runbook tem o copia-e-cola; o loader nunca lança.
|
||||
- **Custo/quota:** lotes de 32, teto de chars por texto, `usage` reportado; corpus inteiro estimado em ~20–30k textos, dezenas de minutos e custo baixo no bge-m3 (relatório imprime tokens/custo).
|
||||
- **Deep-link/contrato de URL:** unit pins dos bytes do modo exato e do round-trip de `relevancia`/`recentes`; e2e do tema+recentes sem loop de redirect.
|
||||
- **Vazamento de gate:** collection nova lida com `user` + `overrideAccess: false` e `read: canReadSpeech`; papel negado não dispara o embedder; cascade do delete com bypass justificado e coberto por int.
|
||||
- **e2e determinístico:** `DEEPINFRA_API_KEY` vazio no webServer do Playwright (o caminho semântico real é coberto por int com resolver injetado, não por e2e).
|
||||
- **Regressão fora do recorte:** gravações/Central/Sollinha não são tocados; `expandSpeechSearchTheme` e seus consumidores ficam; knip garante que nenhum export ficou morto.
|
||||
|
||||
## Aceite de engenharia
|
||||
|
||||
- [ ] Aceite de produto da intenção coberto: tema sem as palavras exatas devolve por sentido; ordenação por proximidade com data como desempate; trecho real sem score; "Termo exato" intacto e alcançável; vazio honesto com saída; degradação com aviso, sem erro nem fantasma; facetas valendo; fala como unidade; sem UI nova de ordenação na Câmara.
|
||||
- [ ] **Aceite de produto verificado pós-deploy com corpus real (não há corpus nos bancos locais):** `pnpm acervo:index --probe "combate à oposição"` e `--probe "impeachment"` no homeserver contra `teqo_1313` após o deploy + indexação, com top-N/scores/cobertura/custo registrados na Issue e no changelog; probes de controle fora do domínio validam o limiar; valor final de `SPEECH_SEMANTIC_MIN_COSINE` registrado.
|
||||
- [ ] Invariantes AGENTS/engineering-standards: `user` + `overrideAccess: false` em toda leitura; `canReadSpeech` reusado sem alargamento; ator negado sem chamada externa (fail-closed); nenhum Consent/PII novo; copy pt-BR e identificadores em inglês; sem score na UI; `openai/*` e `expandSpeechSearchTheme` intocados.
|
||||
- [ ] Migration isolada na Fase 2, aprovada no stop-point, aplicada localmente e tipos gerados (`pnpm generate:types`); cascade da fala coberto por int.
|
||||
- [ ] Testes de domínio previstos (unit de motor/provedor/URL/VM/cards/CLI; int de ranking, limiar, degradação, facetas, sort, evidência, guard) e e2e atualizado (degradado + sort do tema) com `campaignSpeechAcervo` selecionado no PR high-risk.
|
||||
- [ ] Manifesto de e2e com a collection nova; `pnpm gate:fast` verde; PR ready via `pnpm push` (curado, migration = high-risk).
|
||||
- [ ] Runbook `docs/ops/teqo-1313-deploy.md` (§C155/§C225) com o passo de indexação e a entrada de changelog.
|
||||
|
||||
## Decisões de engenharia
|
||||
|
||||
### D1 — Motor e provider
|
||||
|
||||
**Opções:** A) embeddings multilingual via DeepInfra em memória | B) expansão lexical/LLM como motor | C) pgvector/serviço vetorial dedicado | D) modelo local sem provider.
|
||||
**Recomendação:** A — reusa `DEEPINFRA_API_KEY` e o padrão `fetch`+timeout+`null` do ASR, sem contrato/secret novo; o motor é cosine em memória no app (a imagem não tem `vector`); a query é 1 chamada e o índice é derivado.
|
||||
**Rejeitadas:** B porque é o comportamento rejeitado pelo dono; C porque pgvector exige trocar a imagem de produção e ANN não se paga em ~1k falas; D porque exigiria runtime de modelo no container (infra/custo) sem ganho comprovado.
|
||||
|
||||
### D2 — Forma do índice: unidade e encoding
|
||||
|
||||
**Opções:** A) vetor da fala + vetor por trecho (segmento de ASR; janelas do texto quando não há segmento), JSON de números | B) só chunk, ranking pelo melhor chunk | C) só fala, sem vetor de trecho | D) trechos com texto duplicado no índice.
|
||||
**Recomendação:** A — o vetor da fala (média L2-normalizada dos trechos, computada na indexação) dá um ranking estável e barato; os vetores de trecho só são carregados para as 25 falas da página e entregam o "Trecho mais próximo do tema" real; JSON v1 é simples e auditável, com gatilho explícito para base64 Float32 (4× menor) e/ou int8 (10×) se o tamanho/latência doerem (o relatório do CLI imprime linhas/bytes estimados).
|
||||
**Rejeitadas:** B porque supervaloriza uma frase solta e multiplica o payload de rank; C porque sem trecho não há evidência real (e o design exige); D porque duplica transcrição (o texto da janela é recomputado de `officialTranscript`/`summary` pela MESMA função pura do indexador).
|
||||
|
||||
### D3 — Onde o índice mora
|
||||
|
||||
**Opções:** A) collection oculta nova `speechEmbedding` | B) campos JSON ocultos em `speech` (fala) e `speechSegment` (trechos) | C) storage externo/artefato em arquivo.
|
||||
**Recomendação:** A — o `speech` fica leve, o índice é reconstruível/apagável sem tocar no acervo, o acesso é um predicado só (`canReadSpeech`) e o cascade do delete é explícito (FK de relationship é `ON DELETE SET NULL` sobre coluna NOT NULL, então filhos antes do pai).
|
||||
**Rejeitadas:** B porque incha a tabela principal, faz cada leitura do acervo carregar TOAST de vetores e mistura rebuild do índice com escrita das falas; C porque cria artefato volátil fora do banco, sem access control nem cascade (o banco local de dev/test também precisa do índice semeado nos int).
|
||||
|
||||
### D4 — Candidatos e ordenação no modo tema
|
||||
|
||||
**Opções:** A) candidatos = `where` de facetas/origem apenas; ranking em memória; sem pernas textuais e sem C174 | B) união de resultados semânticos + matches literais do `q` | C) híbrido com boost lexical.
|
||||
**Recomendação:** A — é a promessa do modo ("ordem por proximidade"); o C174 (corte por título) e o `LIKE` só valem onde já valiam (modo exato/degradado), e os matches literais que importarem aparecem naturalmente se passarem no limiar, com o selo "Termo exato" coexistindo.
|
||||
**Rejeitadas:** B/C porque reintroduzem a dependência de termo que motivou o item, quebram "data apenas como desempate" (resultado lexical sem score) e transformam o limiar em ficção.
|
||||
|
||||
### D5 — Contrato de URL do `sort`
|
||||
|
||||
**Opções:** A) `sort` ciente de `mode` no domínio de falas (`relevancia` omitido no tema; `recentes` serializado só no tema; bytes do exato intactos) | B) `relevancia` no vocabulário compartilhado | C) novo parâmetro `order`.
|
||||
**Recomendação:** A — `relevancia` não vaza para gravações/Central; `sort=recentes` no tema é estado real e canônico; o modo exato continua byte-idêntico (nenhum deep-link muda); o `AcervoSortSelect` recebe o conjunto de opções por prop e o mapeamento "valor do select → estado" fica puro e testado.
|
||||
**Rejeitadas:** B porque acrescentaria "Mais relevantes" ao seletor das gravações (fora do recorte); C porque duplica o vocabulário e cria duas verdades para ordenar.
|
||||
|
||||
### D6 — Limiar e vazio honesto
|
||||
|
||||
**Opções:** A) limiar global fixo `SPEECH_SEMANTIC_MIN_COSINE = 0.4`, calibrado por `--probe` | B) sem limiar (mostra sempre top-25) | C) limiar adaptativo por distribuição da consulta | D) expor score/percentual.
|
||||
**Recomendação:** A — valor inicial 0.40 sobre vetores L2-normalizados (bge-m3); calibrar no pós-deploy com os dois temas do aceite (falso-vazio: se o topo ficar vazio, o limiar está alto) e com 3–4 temas de controle fora do domínio (falso-cheio: se controle entrar no topo, o limiar está baixo); o valor final é uma constante de uma linha, registrada com a evidência.
|
||||
**Rejeitadas:** B porque viola "nunca resultado 'parecido' só para preencher a tela"; C porque é imprevisível/inexplicável e não testável; D porque é proibido pela intenção (número cru engana o assessor).
|
||||
|
||||
### D7 — Mecanismo de degradação
|
||||
|
||||
**Opções:** A) busca lexical literal de hoje + aviso discreto | B) degradar mantendo a expansão lexical do C192 | C) esconder o modo tema | D) erro pedindo retry.
|
||||
**Recomendação:** A — `themeUnavailable` quando o embed da query `null` ou `indexedCandidates === 0`; o caminho é o `buildSpeechListWhere` literal (sem `themeTerms`) + C174, ordenado por data; a expansão lexical sai do acervo de falas (`themeTerms` removido do builder e do VM) mas sobrevive intacta para gravações/Central/conteúdo.
|
||||
**Rejeitadas:** B porque mantém o motor rejeitado como rede de segurança e mistura proveniências; C porque some com a feature em silêncio; D porque é o "erro na cara" que a intenção proíbe.
|
||||
|
||||
### D8 — Evidência e contrato do card
|
||||
|
||||
**Opções:** A) melhor trecho real (segmento/janela) por fala da página, sem destaque, `semanticMatch` no VM | B) manter `themeMatchTerm`/destaque lexical no modo tema | C) justificativa gerada por LLM.
|
||||
**Recomendação:** A — o trecho vem do vetor de trecho (cosine com a query), é texto real do acervo, sem `<mark>` e sem score; o rótulo é "Trecho mais próximo do tema"; "Tema" e "Termo exato" coexistem; `watchHref` leva o `?t=` ao trecho de evidência quando conhecido.
|
||||
**Rejeitadas:** B porque destaca um termo que não é o motor (mentira de proveniência) e o design o removeu; C por alucinação e imprevisibilidade.
|
||||
|
||||
### D9 — Indexação: CLI separado vs acoplado ao import
|
||||
|
||||
**Opções:** A) CLI manual `pnpm acervo:index` (idempotente por hash+modelo, `--dry-run`/`--limit`/`--source`/`--probe`), encadeado no runbook após os imports | B) indexar dentro de `import-web-speeches.mjs`/`import-camara` | C) job assíncrono no app.
|
||||
**Recomendação:** A — um dono só para batching, hash, skip, relatório e probe; os imports seguem manuais e idempotentes; o `--probe` é o instrumento de calibração e de verificação pós-deploy; escrita com `ACERVO_INDEX_CONFIRM=1` e relatório honesto.
|
||||
**Rejeitadas:** B porque duplicaria a lógica em dois scripts (e o backfill da Câmara está congelado); C porque é o rabbit hole de reindexação em tempo real. Gatilho de revisitação: se a defasagem do índice doer, o import chama a mesma função do CLI (sem segundo dono).
|
||||
|
||||
### D10 — Injeção do embedding da query (testes offline)
|
||||
|
||||
**Opções:** A) 4º parâmetro `embedQuery` injetável nas duas loaders (precedente `expandTheme`/`SpeechVideoStartResolver`) | B) mock global do módulo | C) testes só com chave real.
|
||||
**Recomendação:** A — int tests determinísticos semeiam `speechEmbedding` e injetam o vetor da query, exercitando ranking/limiar/degradação sem rede; o default é `embedSpeechQuery`; o e2e apenas cobre o degradado (chave anulada no Playwright).
|
||||
**Rejeitadas:** B porque acopla o teste ao bundler e esconde o contrato; C porque depende de rede/provedor e não roda no CI.
|
||||
|
||||
### D11 — Expansão lexical fora do caminho de falas (sem deletar o dono)
|
||||
|
||||
**Opções:** A) remover `themeTerms` do builder/VM de falas e deixar `expandSpeechSearchTheme` intacto para os outros corpus | B) deletar `expandSpeechSearchTheme` | C) manter o parâmetro morto nos filtros.
|
||||
**Recomendação:** A — o acervo de falas deixa de chamar a expansão; gravações (`recordingPageData`/`expandRecordingSearchTheme`), Central (`expandContentPieceSearchTheme`) e conteúdo continuam usando o mesmo dono; knip garante que nada exportado ficou sem consumidor.
|
||||
**Rejeitadas:** B porque gravações/Central/conteúdo dependem do módulo (a intenção manda manter a expansão deles); C porque superfície morta mente sobre o mecanismo.
|
||||
|
||||
## Self-score decision-quality
|
||||
|
||||
1. **Decisões caras com rejeitadas:** 5/5 — motor/provider, forma e moradia do índice, candidatos/ordenação, contrato de URL, limiar, degradação, evidência, CLI e injeção têm opções/recomendação/rejeitadas registradas.
|
||||
2. **Cabe no appetite:** 4/5 — ~2–3 dias no limite superior: os módulos são pequenos e o UI é port classe-a-classe, mas schema+CLI+rework dos dois loaders concentram o esforço; o stop-point da migration não conta como implementação.
|
||||
3. **Rabbit holes nomeados:** 5/5 — pgvector/infra vetorial, reindexação em tempo real, expansão como motor, gravações/Central/Sollinha, score/justificativa e quantização deferida com gatilho estão cortados.
|
||||
4. **Depth check (reusa shells/helpers):** 5/5 — `CampaignListEmptyState`, `AcervoSortSelect`, `CampaignThemeRetryButton`, `SpeechThemeFallbackNotice`, `SpeechResultCard`/`WebSpeechResultCard`, `loadSegmentsForSpeeches`, `speechHighlight`, `campaignListUrl`, `assertLocalDatabase`/`assertWriteConfirm`, o padrão `fetch`+timeout+null do DeepInfra e a injeção de resolver.
|
||||
5. **Intenção permanece satisfeita:** 5/5 — outcome, anti-goals e guardrails de acesso intactos; a engenharia troca o mecanismo sem reescrever o produto.
|
||||
|
||||
**Total: 5/5** (gate ≥4 satisfeito).
|
||||
|
|
@ -91,6 +91,7 @@
|
|||
"camara:pilot": "cross-env NODE_OPTIONS=\"--no-deprecation --import=tsx/esm\" node scripts/pilot-camara-speeches.mjs",
|
||||
"camara:import": "cross-env NODE_OPTIONS=\"--no-deprecation --import=tsx/esm --import=./scripts/seed-loader.mjs\" node scripts/import-camara-speeches.mjs",
|
||||
"falas-web:import": "cross-env NODE_OPTIONS=\"--no-deprecation --import=tsx/esm --import=./scripts/seed-loader.mjs\" node scripts/import-web-speeches.mjs",
|
||||
"acervo:index": "cross-env NODE_OPTIONS=\"--no-deprecation --import=tsx/esm --import=./scripts/seed-loader.mjs\" node scripts/index-speeches.mjs",
|
||||
"recordings:classify": "cross-env NODE_OPTIONS=\"--no-deprecation --import=tsx/esm --import=./scripts/seed-loader.mjs\" node scripts/backfill-recording-facets.mjs",
|
||||
"typecheck": "tsc --noEmit",
|
||||
"typecheck:tsc6": "tsc6 --noEmit",
|
||||
|
|
|
|||
|
|
@ -283,8 +283,10 @@ export default defineConfig({
|
|||
* C192 — blank the DeepSeek key so the acervo's theme expansion always
|
||||
* degrades to the literal search (never the network) and the degraded
|
||||
* state stays deterministic, same principle as the Google test key.
|
||||
* C229 — the same for the embeddings provider of the speech sources.
|
||||
*/
|
||||
DEEPSEEK_API_KEY: '',
|
||||
DEEPINFRA_API_KEY: '',
|
||||
PAYLOAD_SECRET: process.env.PAYLOAD_SECRET ?? 'test-only-secret-not-used-in-production',
|
||||
// Prod mode serves a production build where `getCampaignInviteBaseURL`
|
||||
// fails closed unless NEXT_PUBLIC_SITE_URL is an HTTPS public DNS name
|
||||
|
|
|
|||
224
scripts/index-speeches.mjs
Normal file
224
scripts/index-speeches.mjs
Normal file
|
|
@ -0,0 +1,224 @@
|
|||
/**
|
||||
* C229 — builds/updates the semantic index of the speech acervo (`pnpm
|
||||
* acervo:index`). It plans the retrieval units of every speech (ASR segments
|
||||
* or text windows), embeds only what changed — idempotent by content hash +
|
||||
* model — writes the vectors in one transaction per speech and prints an honest
|
||||
* report (indexed/skipped/failed/units/tokens/cost). `--probe "<tema>"` is the
|
||||
* read-only calibration: it embeds one query and prints the top-N indexed
|
||||
* speeches with their cosine, without writing to the database.
|
||||
*
|
||||
* The corpus and the provider live on the homeserver: run this after the Câmara
|
||||
* backfill or a web round (see `docs/ops/teqo-1313-deploy.md` §C155/§C225), and
|
||||
* pass `ACERVO_INDEX_CONFIRM=1` when the target is not provably local (the
|
||||
* homeserver runs with `NODE_ENV=production` behind the socat proxy).
|
||||
*
|
||||
* Usage:
|
||||
* pnpm acervo:index --dry-run
|
||||
* pnpm acervo:index --source web --limit 20
|
||||
* ACERVO_INDEX_CONFIRM=1 pnpm acervo:index --source all
|
||||
* ACERVO_INDEX_CONFIRM=1 pnpm acervo:index --force
|
||||
* pnpm acervo:index --probe "combate à oposição" --top 10
|
||||
*/
|
||||
import { mkdir, writeFile } from 'node:fs/promises'
|
||||
import { dirname, join } from 'node:path'
|
||||
|
||||
import { getPayload } from 'payload'
|
||||
|
||||
import { assertLocalDatabase } from './assert-local-database.mjs'
|
||||
import { assertWriteConfirm, databaseTarget, dieWithLabel, loadCliEnv } from './lib/cli.mjs'
|
||||
import {
|
||||
ACERVO_INDEX_CONFIRM_FLAG,
|
||||
formatSpeechIndexProbe,
|
||||
formatSpeechIndexReport,
|
||||
parseSpeechIndexArgs,
|
||||
speechIndexCostUsd,
|
||||
speechIndexEstimatedBytes,
|
||||
speechIndexReportStamp,
|
||||
} from './lib/speechIndex.mjs'
|
||||
|
||||
loadCliEnv()
|
||||
|
||||
const die = dieWithLabel('acervo:index')
|
||||
|
||||
const config = (await import('../src/payload.config.ts')).default
|
||||
const {
|
||||
DEEPINFRA_EMBED_COST_PER_MILLION_TOKENS_USD,
|
||||
DEEPINFRA_EMBED_DIMENSIONS,
|
||||
DEEPINFRA_EMBED_MODEL,
|
||||
embedSpeechQuery,
|
||||
} = await import('../src/utilities/ai/deepInfraEmbed.ts')
|
||||
const { indexSpeechSources, planSpeechIndexBatch, probeSpeechIndex, previewSpeechIndex } =
|
||||
await import('../src/utilities/speech/speechEmbeddingIndex.ts')
|
||||
|
||||
const HELP = `
|
||||
Uso: pnpm acervo:index [opções]
|
||||
|
||||
Constrói/atualiza o índice vetorial do acervo de falas (Câmara + internet) e
|
||||
relata o que entrou. Idempotente por hash de conteúdo + modelo: reexecutar só
|
||||
paga o que mudou. Após os imports (C155/C225), rode este passo.
|
||||
|
||||
Opções:
|
||||
--source <all|camara|web> recorte do acervo (default all)
|
||||
--limit <n> processa no máximo n falas
|
||||
--dry-run planeja sem chamar o provedor nem escrever
|
||||
--force reindexa mesmo com hash atual
|
||||
--probe <tema> read-only: top-N por similaridade (calibração)
|
||||
--top <n> quantos no probe (default 10)
|
||||
--out <dir> diretório dos relatórios (default data/acervo-index)
|
||||
--help esta ajuda
|
||||
|
||||
Escrita em alvo não-local, com NODE_ENV=production ou com ALLOW_REMOTE_DB exige
|
||||
${ACERVO_INDEX_CONFIRM_FLAG}=1. O probe é read-only e não exige a flag.
|
||||
`
|
||||
|
||||
/** Every speech of the recorte with its segments, in one pass per collection. */
|
||||
const loadSources = async (payload, { source, limit }) => {
|
||||
const where = source === 'all' ? {} : { origin: { equals: source } }
|
||||
const speeches = await payload.find({
|
||||
collection: 'speech',
|
||||
where,
|
||||
depth: 0,
|
||||
...(limit ? { limit, page: 1 } : { limit: 0, pagination: false }),
|
||||
sort: 'speechAt',
|
||||
select: { sourceKey: true, officialTranscript: true, summary: true, searchText: true },
|
||||
// Intentional bypass: the index CLI is a trusted actor with no session.
|
||||
overrideAccess: true,
|
||||
})
|
||||
const ids = speeches.docs.map((speech) => speech.id)
|
||||
|
||||
const segmentsBySpeech = new Map()
|
||||
if (ids.length > 0) {
|
||||
const segments = await payload.find({
|
||||
collection: 'speechSegment',
|
||||
where: { speech: { in: ids } },
|
||||
depth: 0,
|
||||
limit: 0,
|
||||
pagination: false,
|
||||
sort: 'order',
|
||||
select: { speech: true, order: true, startSeconds: true, text: true },
|
||||
// Intentional bypass: same trusted read.
|
||||
overrideAccess: true,
|
||||
})
|
||||
for (const segment of segments.docs) {
|
||||
const speechId = typeof segment.speech === 'number' ? segment.speech : segment.speech?.id
|
||||
const list = segmentsBySpeech.get(speechId) ?? []
|
||||
list.push({
|
||||
order: segment.order,
|
||||
startSeconds: segment.startSeconds,
|
||||
text: segment.text,
|
||||
})
|
||||
segmentsBySpeech.set(speechId, list)
|
||||
}
|
||||
}
|
||||
|
||||
const sources = speeches.docs.map((speech) => ({
|
||||
id: speech.id,
|
||||
sourceKey: speech.sourceKey,
|
||||
officialTranscript: speech.officialTranscript,
|
||||
summary: speech.summary,
|
||||
searchText: speech.searchText,
|
||||
segments: segmentsBySpeech.get(speech.id) ?? [],
|
||||
}))
|
||||
return sources
|
||||
}
|
||||
|
||||
const writeReport = async (options, report) => {
|
||||
const reportPath = join(
|
||||
options.out,
|
||||
'reports',
|
||||
`acervo-index-${speechIndexReportStamp(report.runAt)}.json`,
|
||||
)
|
||||
await mkdir(dirname(reportPath), { recursive: true })
|
||||
await writeFile(reportPath, JSON.stringify(report, null, 2))
|
||||
return reportPath
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const options = parseSpeechIndexArgs(process.argv.slice(2))
|
||||
if (options.help) {
|
||||
console.log(HELP)
|
||||
process.exit(0)
|
||||
}
|
||||
|
||||
assertLocalDatabase(
|
||||
'acervo:index',
|
||||
'O índice escreve no acervo; produção exige ' + `${ACERVO_INDEX_CONFIRM_FLAG}=1.`,
|
||||
)
|
||||
if (!options.probe && !options.dryRun) {
|
||||
assertWriteConfirm({
|
||||
label: 'acervo:index',
|
||||
flag: ACERVO_INDEX_CONFIRM_FLAG,
|
||||
command: 'pnpm acervo:index',
|
||||
})
|
||||
}
|
||||
|
||||
const payload = await getPayload({ config })
|
||||
|
||||
if (options.probe) {
|
||||
const queryVector = await embedSpeechQuery(options.probe)
|
||||
if (!queryVector) {
|
||||
die('não foi possível vetorizar o tema (DEEPINFRA_API_KEY ausente ou provedor fora).')
|
||||
}
|
||||
const hits = await probeSpeechIndex(payload, queryVector, { limit: options.top })
|
||||
for (const line of formatSpeechIndexProbe(options.probe, hits)) console.log(line)
|
||||
process.exit(0)
|
||||
}
|
||||
|
||||
const runAt = new Date().toISOString()
|
||||
const startedAt = Date.now()
|
||||
const sources = await loadSources(payload, options)
|
||||
console.log(
|
||||
`[acervo:index] alvo: ${databaseTarget()} | modo: ${options.dryRun ? 'dry-run (read-only)' : 'index'} | fonte: ${options.source} | falas no recorte: ${sources.length}`,
|
||||
)
|
||||
|
||||
if (options.dryRun) {
|
||||
const batch = await planSpeechIndexBatch(payload, sources, { force: options.force })
|
||||
const preview = previewSpeechIndex(batch, sources.length)
|
||||
const report = {
|
||||
runAt,
|
||||
mode: 'dry-run',
|
||||
target: databaseTarget(),
|
||||
source: options.source,
|
||||
model: DEEPINFRA_EMBED_MODEL,
|
||||
...preview,
|
||||
estimatedBytes: speechIndexEstimatedBytes(preview.units, DEEPINFRA_EMBED_DIMENSIONS),
|
||||
durationMs: Date.now() - startedAt,
|
||||
}
|
||||
const reportPath = await writeReport(options, report)
|
||||
for (const line of formatSpeechIndexReport(report)) console.log(line)
|
||||
console.log(`\n[acervo:index] relatório JSON: ${reportPath}`)
|
||||
process.exit(0)
|
||||
}
|
||||
|
||||
const result = await indexSpeechSources(payload, sources, { force: options.force })
|
||||
const report = {
|
||||
runAt,
|
||||
mode: 'index',
|
||||
target: databaseTarget(),
|
||||
source: options.source,
|
||||
model: DEEPINFRA_EMBED_MODEL,
|
||||
speeches: sources.length,
|
||||
upToDate: result.upToDate,
|
||||
skippedNoText: result.skippedNoText,
|
||||
toIndex: result.indexed + result.failed,
|
||||
indexed: result.indexed,
|
||||
failed: result.failed,
|
||||
units: result.units,
|
||||
estimatedBytes: speechIndexEstimatedBytes(result.units, DEEPINFRA_EMBED_DIMENSIONS),
|
||||
promptTokens: result.promptTokens,
|
||||
costUsd: speechIndexCostUsd(result.promptTokens, DEEPINFRA_EMBED_COST_PER_MILLION_TOKENS_USD),
|
||||
durationMs: Date.now() - startedAt,
|
||||
}
|
||||
const reportPath = await writeReport(options, report)
|
||||
for (const line of formatSpeechIndexReport(report)) console.log(line)
|
||||
console.log(`\n[acervo:index] relatório JSON: ${reportPath}`)
|
||||
|
||||
if (result.failed > 0 && result.indexed === 0) {
|
||||
die('nenhuma fala foi indexada — falhas listadas no relatório.')
|
||||
}
|
||||
process.exit(0)
|
||||
}
|
||||
|
||||
main().catch((error) => {
|
||||
die(error?.message || String(error))
|
||||
})
|
||||
|
|
@ -499,6 +499,8 @@ export const E2E_AFFECTED_MANIFEST = [
|
|||
'src/lib/webSpeech',
|
||||
'src/collections/Speech.ts',
|
||||
'src/collections/InternetSpeechMedia.ts',
|
||||
// C229 — the semantic index collection/engine the theme mode reads.
|
||||
'src/collections/SpeechEmbedding.ts',
|
||||
// C216 — the source and ordering vocabularies shared by the acervo.
|
||||
'src/lib/acervoSource',
|
||||
'src/lib/acervoListSort',
|
||||
|
|
|
|||
116
scripts/lib/speechIndex.mjs
Normal file
116
scripts/lib/speechIndex.mjs
Normal file
|
|
@ -0,0 +1,116 @@
|
|||
/**
|
||||
* C229 — pure helpers of `pnpm acervo:index`: argument parsing, the report
|
||||
* shape/cost math and the `--probe` rendering. The DB/embedding work lives in
|
||||
* `src/utilities/speech/speechEmbeddingIndex.ts`; this module is pinned by
|
||||
* unit tests so a flag or a report line cannot drift silently.
|
||||
*/
|
||||
|
||||
export const ACERVO_INDEX_CONFIRM_FLAG = 'ACERVO_INDEX_CONFIRM'
|
||||
const DEFAULT_OUT_DIR = 'data/acervo-index'
|
||||
const SPEECH_INDEX_SOURCES = ['all', 'camara', 'web']
|
||||
|
||||
/**
|
||||
* @param {string[]} argv
|
||||
*/
|
||||
export const parseSpeechIndexArgs = (argv) => {
|
||||
const options = {
|
||||
source: 'all',
|
||||
limit: null,
|
||||
dryRun: false,
|
||||
force: false,
|
||||
probe: null,
|
||||
top: 10,
|
||||
out: DEFAULT_OUT_DIR,
|
||||
help: false,
|
||||
}
|
||||
|
||||
const fail = (message) => {
|
||||
throw new Error(message)
|
||||
}
|
||||
|
||||
for (let index = 0; index < argv.length; index += 1) {
|
||||
const arg = argv[index]
|
||||
const value = () => {
|
||||
const next = argv[++index]
|
||||
if (next === undefined || next.startsWith('--')) fail(`faltou valor para ${arg}.`)
|
||||
return next
|
||||
}
|
||||
if (arg === '--help' || arg === '-h') options.help = true
|
||||
else if (arg === '--source') options.source = value()
|
||||
else if (arg === '--limit') options.limit = Number(value())
|
||||
else if (arg === '--dry-run') options.dryRun = true
|
||||
else if (arg === '--force') options.force = true
|
||||
else if (arg === '--probe') options.probe = value()
|
||||
else if (arg === '--top') options.top = Number(value())
|
||||
else if (arg === '--out') options.out = value()
|
||||
else fail(`argumento desconhecido: ${arg}`)
|
||||
}
|
||||
|
||||
if (options.help) return options
|
||||
if (!SPEECH_INDEX_SOURCES.includes(options.source)) {
|
||||
fail(`--source deve ser um de: ${SPEECH_INDEX_SOURCES.join(', ')}.`)
|
||||
}
|
||||
if (options.limit !== null && (!Number.isInteger(options.limit) || options.limit < 1)) {
|
||||
fail('--limit deve ser >= 1.')
|
||||
}
|
||||
if (!Number.isInteger(options.top) || options.top < 1) fail('--top deve ser >= 1.')
|
||||
if (options.out.includes('..')) fail('--out não pode escapar do diretório do repo (sem "..").')
|
||||
if (options.probe && (options.dryRun || options.force || options.limit !== null)) {
|
||||
fail('--probe é read-only e não combina com --dry-run/--force/--limit.')
|
||||
}
|
||||
if (options.top !== 10 && !options.probe) fail('--top só vale com --probe.')
|
||||
return options
|
||||
}
|
||||
|
||||
/** Filesystem-safe stamp of the run instant (same shape as the other reports). */
|
||||
export const speechIndexReportStamp = (iso) => iso.replace(/[:.]/g, '-')
|
||||
|
||||
export const speechIndexCostUsd = (promptTokens, pricePerMillionTokensUsd) =>
|
||||
(promptTokens / 1_000_000) * pricePerMillionTokensUsd
|
||||
|
||||
/** Rough JSON cost of the vectors: units × dims × ~8 bytes per number. */
|
||||
export const speechIndexEstimatedBytes = (units, dimensions) => units * dimensions * 8
|
||||
|
||||
/**
|
||||
* @param {{
|
||||
* target: string, mode: 'index' | 'dry-run', source: string,
|
||||
* speeches: number, upToDate: number, skippedNoText: number, toIndex: number,
|
||||
* units: number, estimatedBytes?: number, indexed?: number, failed?: number,
|
||||
* promptTokens?: number, costUsd?: number, durationMs: number,
|
||||
* }} report
|
||||
* @returns {string[]}
|
||||
*/
|
||||
export const formatSpeechIndexReport = (report) => {
|
||||
const bytes =
|
||||
typeof report.estimatedBytes === 'number'
|
||||
? ` | bytes estimados: ${(report.estimatedBytes / 1_000_000).toFixed(1)} MB`
|
||||
: ''
|
||||
const lines = [
|
||||
`[acervo:index] alvo: ${report.target} | modo: ${report.mode} | fonte: ${report.source}`,
|
||||
`falas: ${report.speeches} | atualizadas: ${report.upToDate} | sem texto: ${report.skippedNoText} | a indexar: ${report.toIndex}`,
|
||||
]
|
||||
|
||||
if (report.mode === 'index') {
|
||||
lines.push(
|
||||
`indexadas: ${report.indexed ?? 0} | falhas: ${report.failed ?? 0} | trechos vetorizados: ${report.units}${bytes} | tokens: ${report.promptTokens ?? 0} | custo estimado: US$ ${(report.costUsd ?? 0).toFixed(4)}`,
|
||||
)
|
||||
} else {
|
||||
lines.push(`trechos a vetorizar: ${report.units}${bytes}`)
|
||||
}
|
||||
|
||||
lines.push(`duração: ${(report.durationMs / 1000).toFixed(1)}s`)
|
||||
return lines
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {string} term
|
||||
* @param {{ score: number, speechAt: string, sourceKey: string, origin: string }[]} hits
|
||||
* @returns {string[]}
|
||||
*/
|
||||
export const formatSpeechIndexProbe = (term, hits) => [
|
||||
`[acervo:index] probe "${term}" — top ${hits.length}`,
|
||||
...hits.map(
|
||||
(hit, index) =>
|
||||
`${index + 1}. ${hit.score.toFixed(4)} · ${hit.speechAt} · ${hit.origin} · ${hit.sourceKey}`,
|
||||
),
|
||||
]
|
||||
|
|
@ -100,6 +100,7 @@ export const SCRIPTS_SPEC_PINNED = [
|
|||
'scripts/lib/reportText.mjs',
|
||||
'scripts/lib/seed-minimal-manifest.mjs',
|
||||
'scripts/lib/sollaCeuciSalvadorMetrics.mjs',
|
||||
'scripts/lib/speechIndex.mjs',
|
||||
'scripts/lib/staging-test-account.mjs',
|
||||
'scripts/lib/test-affected-core.mjs',
|
||||
'scripts/lib/testing-audit-metrics-core.mjs',
|
||||
|
|
|
|||
|
|
@ -61,7 +61,7 @@ type SpeechAcervoPageProps = {
|
|||
* (the C216 web speeches).
|
||||
*/
|
||||
const AcervoHeader = ({ source }: { source: AcervoSource }) => (
|
||||
<div className="flex flex-col gap-4 pt-4 md:pt-0">
|
||||
<div className="flex flex-col gap-2 pt-4 max-md:-mb-6 md:gap-4 md:pt-0">
|
||||
{/* Mobile keeps only the switcher row: the shell chrome already carries the
|
||||
title and the action joins the toggle (approved scene 6). */}
|
||||
<div className="flex flex-wrap items-end justify-between gap-4 max-md:hidden">
|
||||
|
|
@ -81,12 +81,14 @@ const AcervoHeader = ({ source }: { source: AcervoSource }) => (
|
|||
<RecordingUploadDialog />
|
||||
</div>
|
||||
</div>
|
||||
<div className="flex flex-wrap items-center justify-between gap-3">
|
||||
<AcervoSourceToggle source={source} />
|
||||
<div className="flex items-center justify-between gap-2 md:gap-3">
|
||||
<div className="min-w-0 flex-1 md:flex-none">
|
||||
<AcervoSourceToggle source={source} />
|
||||
</div>
|
||||
<RecordingUploadDialog
|
||||
triggerLabel=""
|
||||
triggerAriaLabel="Enviar gravação"
|
||||
triggerClassName="min-h-10 md:hidden"
|
||||
triggerClassName="min-h-9 md:hidden"
|
||||
/>
|
||||
</div>
|
||||
</div>
|
||||
|
|
@ -97,11 +99,14 @@ const AcervoHeader = ({ source }: { source: AcervoSource }) => (
|
|||
* term / theme / idle) and the hint line of `mode=tema`. The caller owns the
|
||||
* right side (`controls`: the Câmara leaves the retry button bare, the other
|
||||
* two wrap it with their sort select) and whether the row renders at all.
|
||||
* C229 — `semanticTheme` swaps the hint of the speech sources for the design
|
||||
* copy of the sense engine; the recordings keep their lexical wording.
|
||||
*/
|
||||
const AcervoResultsHeading = ({
|
||||
themeMode,
|
||||
themeActive,
|
||||
themeUnavailable,
|
||||
semanticTheme = false,
|
||||
idleTitle,
|
||||
itemNoun,
|
||||
controls,
|
||||
|
|
@ -109,6 +114,7 @@ const AcervoResultsHeading = ({
|
|||
themeMode: boolean
|
||||
themeActive: boolean
|
||||
themeUnavailable: boolean
|
||||
semanticTheme?: boolean
|
||||
idleTitle: string
|
||||
itemNoun: 'fala' | 'gravação'
|
||||
controls: ReactNode
|
||||
|
|
@ -123,13 +129,32 @@ const AcervoResultsHeading = ({
|
|||
: idleTitle}
|
||||
</h2>
|
||||
{themeMode ? (
|
||||
<p className="mt-0.5 text-xs text-muted-foreground">
|
||||
{themeUnavailable
|
||||
? 'Comportamento atual do acervo.'
|
||||
: themeActive
|
||||
? `Confira o indício em cada ${itemNoun} antes de abrir.`
|
||||
: 'Nenhum termo relacionado foi acrescentado; mostramos a busca literal.'}
|
||||
</p>
|
||||
themeUnavailable ? (
|
||||
<p className="mt-0.5 text-xs text-muted-foreground">
|
||||
{semanticTheme
|
||||
? 'Ordenados por data, como na busca literal.'
|
||||
: 'Comportamento atual do acervo.'}
|
||||
</p>
|
||||
) : themeActive ? (
|
||||
semanticTheme ? (
|
||||
<>
|
||||
<p className="mt-0.5 hidden text-xs text-muted-foreground md:block">
|
||||
Do mais próximo ao menos próximo do sentido buscado.
|
||||
</p>
|
||||
<p className="mt-0.5 text-xs text-muted-foreground md:hidden">
|
||||
Ordenados pela proximidade de sentido.
|
||||
</p>
|
||||
</>
|
||||
) : (
|
||||
<p className="mt-0.5 text-xs text-muted-foreground">
|
||||
Confira o indício em cada {itemNoun} antes de abrir.
|
||||
</p>
|
||||
)
|
||||
) : (
|
||||
<p className="mt-0.5 text-xs text-muted-foreground">
|
||||
Nenhum termo relacionado foi acrescentado; mostramos a busca literal.
|
||||
</p>
|
||||
)
|
||||
) : null}
|
||||
</div>
|
||||
{controls}
|
||||
|
|
@ -149,9 +174,9 @@ const AcervoResultsControls = ({
|
|||
themeUnavailable: boolean
|
||||
hint: string
|
||||
}) => (
|
||||
<div className="flex flex-wrap items-end gap-2">
|
||||
<div className="flex w-full flex-wrap items-end gap-2 md:w-auto">
|
||||
{themeUnavailable ? <CampaignThemeRetryButton /> : null}
|
||||
<AcervoSortSelect state={state} hint={hint} />
|
||||
<AcervoSortSelect state={state} hint={hint} themeUnavailable={themeUnavailable} />
|
||||
</div>
|
||||
)
|
||||
|
||||
|
|
@ -293,6 +318,7 @@ const WebSpeechesSource = ({
|
|||
themeMode={themeMode}
|
||||
themeActive={themeActive}
|
||||
themeUnavailable={data.themeUnavailable}
|
||||
semanticTheme
|
||||
idleTitle="Resultados encontrados"
|
||||
itemNoun="fala"
|
||||
controls={
|
||||
|
|
@ -310,17 +336,18 @@ const WebSpeechesSource = ({
|
|||
<CampaignListEmptyState
|
||||
className="border-solid"
|
||||
mediaClassName="rounded-none bg-transparent"
|
||||
headerClassName={themeActive ? 'max-w-lg [text-wrap:wrap]' : undefined}
|
||||
icon={SearchXIcon}
|
||||
title={
|
||||
themeActive
|
||||
? 'Nenhuma fala encontrada para este tema'
|
||||
? 'Nenhuma fala combina bem com esse tema'
|
||||
: hasFilters
|
||||
? 'Nenhuma fala encontrada'
|
||||
: 'Nenhuma fala da internet catalogada ainda'
|
||||
}
|
||||
description={
|
||||
themeActive
|
||||
? 'Não encontramos uma fala que corresponda ao sentido desta busca com os filtros atuais. Não vamos preencher a lista com resultados pouco relacionados.'
|
||||
? 'Não encontramos uma fala próxima o suficiente do sentido buscado com os filtros atuais. Não vamos completar a lista com resultados pouco relacionados.'
|
||||
: hasFilters
|
||||
? 'Não encontramos uma fala para esta busca com os filtros atuais. Tente outro termo ou remova um filtro.'
|
||||
: 'As falas encontradas na internet aparecem aqui depois que a ingestão as catalogar.'
|
||||
|
|
@ -334,7 +361,7 @@ const WebSpeechesSource = ({
|
|||
Usar termo exato
|
||||
</Link>
|
||||
</Button>
|
||||
<Button asChild variant="ghost" className="min-h-11">
|
||||
<Button asChild variant="ghost" className="min-h-11 text-muted-foreground">
|
||||
<Link href={buildAcervoSourceHref('internet')}>Limpar filtros</Link>
|
||||
</Button>
|
||||
</div>
|
||||
|
|
@ -387,6 +414,7 @@ const CamaraSource = ({ data }: { data: Awaited<ReturnType<typeof loadSpeechAcer
|
|||
themeMode={themeMode}
|
||||
themeActive={themeActive}
|
||||
themeUnavailable={data.themeUnavailable}
|
||||
semanticTheme
|
||||
idleTitle="Resultados"
|
||||
itemNoun="fala"
|
||||
controls={data.themeUnavailable ? <CampaignThemeRetryButton /> : null}
|
||||
|
|
@ -398,19 +426,20 @@ const CamaraSource = ({ data }: { data: Awaited<ReturnType<typeof loadSpeechAcer
|
|||
) : (
|
||||
<CampaignListEmptyState
|
||||
icon={SearchXIcon}
|
||||
headerClassName={themeActive ? 'max-w-lg [text-wrap:wrap]' : undefined}
|
||||
className={themeActive ? 'border-solid' : undefined}
|
||||
mediaClassName={themeActive ? 'size-12 rounded-full' : undefined}
|
||||
contentClassName={themeActive ? 'max-w-none' : undefined}
|
||||
title={
|
||||
themeActive
|
||||
? 'Nenhuma fala encontrada para este tema'
|
||||
? 'Nenhuma fala combina bem com esse tema'
|
||||
: data.state.q
|
||||
? `Nenhuma fala encontrada para "${data.state.q}"`
|
||||
: 'Nenhuma fala encontrada'
|
||||
}
|
||||
description={
|
||||
themeActive
|
||||
? 'Não encontramos uma fala que corresponda ao sentido desta busca com os filtros atuais. Não vamos preencher a lista com resultados pouco relacionados.'
|
||||
? 'Não encontramos uma fala próxima o suficiente do sentido buscado com os filtros atuais. Não vamos completar a lista com resultados pouco relacionados.'
|
||||
: hasFilters
|
||||
? 'Tente outro termo, remova filtros ou busque por um tema.'
|
||||
: 'O acervo ainda não tem falas importadas.'
|
||||
|
|
@ -424,7 +453,7 @@ const CamaraSource = ({ data }: { data: Awaited<ReturnType<typeof loadSpeechAcer
|
|||
Usar termo exato
|
||||
</Link>
|
||||
</Button>
|
||||
<Button asChild variant="ghost" className="min-h-11">
|
||||
<Button asChild variant="ghost" className="min-h-11 text-muted-foreground">
|
||||
<Link href={CAMPAIGN_COMMUNICATION_ACERVO}>Limpar filtros</Link>
|
||||
</Button>
|
||||
</div>
|
||||
|
|
|
|||
|
|
@ -78,6 +78,15 @@ const deleteSpeechAssets: CollectionBeforeDeleteHook = async ({ id, req }) => {
|
|||
.map((media) => (typeof media === 'object' && media !== null ? media.id : media))
|
||||
.filter((mediaId): mediaId is number => typeof mediaId === 'number')
|
||||
|
||||
// C229 — the semantic index is derived; deleting the speech deletes its rows
|
||||
// first (the relationship FK cannot SET NULL over a NOT NULL column).
|
||||
await req.payload.delete({
|
||||
collection: 'speechEmbedding',
|
||||
where: { speech: { equals: id } },
|
||||
req,
|
||||
// Intentional bypass: same cascade over derived vectors.
|
||||
overrideAccess: true,
|
||||
})
|
||||
await req.payload.delete({
|
||||
collection: 'speechSegment',
|
||||
where: { speech: { equals: id } },
|
||||
|
|
|
|||
107
src/collections/SpeechEmbedding.ts
Normal file
107
src/collections/SpeechEmbedding.ts
Normal file
|
|
@ -0,0 +1,107 @@
|
|||
import type { CollectionConfig } from 'payload'
|
||||
|
||||
import { canReadSpeech, payloadAdminOnly } from '@/utilities/campaignAccess'
|
||||
|
||||
/**
|
||||
* C229 — the semantic index of the speech acervo, one row per vector. `kind:
|
||||
* 'speech'` is the speech-level vector (element-wise mean of its units, used by
|
||||
* the ranking); `kind: 'segment'` points at an ASR segment by `order` and
|
||||
* `kind: 'window'` at a fixed-size text window of a speech without segments —
|
||||
* both are the real evidence excerpt the card shows ("Trecho mais próximo do
|
||||
* tema"). The vector is L2-normalized at write time and stored as JSON.
|
||||
*
|
||||
* Derived data, never authoritative: the acervo keeps its access gate
|
||||
* (`canReadSpeech`), the import CLIs restore it and `pnpm acervo:index`
|
||||
* rebuilds/stales it by `contentHash` + `model`. Admin-hidden: the product
|
||||
* surface is the catalog UI.
|
||||
*/
|
||||
export const SpeechEmbedding: CollectionConfig = {
|
||||
slug: 'speechEmbedding',
|
||||
labels: {
|
||||
singular: 'Vetor de fala',
|
||||
plural: 'Vetores de fala',
|
||||
},
|
||||
admin: {
|
||||
group: 'Comunicação',
|
||||
hidden: () => true,
|
||||
useAsTitle: 'kind',
|
||||
},
|
||||
access: {
|
||||
create: payloadAdminOnly,
|
||||
read: canReadSpeech,
|
||||
update: payloadAdminOnly,
|
||||
delete: payloadAdminOnly,
|
||||
},
|
||||
fields: [
|
||||
{
|
||||
name: 'speech',
|
||||
type: 'relationship',
|
||||
relationTo: 'speech',
|
||||
label: 'Fala',
|
||||
required: true,
|
||||
index: true,
|
||||
},
|
||||
{
|
||||
name: 'kind',
|
||||
type: 'select',
|
||||
label: 'Unidade',
|
||||
required: true,
|
||||
index: true,
|
||||
options: [
|
||||
{ label: 'Fala', value: 'speech' },
|
||||
{ label: 'Segmento', value: 'segment' },
|
||||
{ label: 'Janela de texto', value: 'window' },
|
||||
],
|
||||
},
|
||||
{
|
||||
name: 'order',
|
||||
type: 'number',
|
||||
label: 'Ordem',
|
||||
admin: {
|
||||
description: 'Posição do trecho na fala; vazio no vetor da fala.',
|
||||
},
|
||||
},
|
||||
{
|
||||
name: 'startSeconds',
|
||||
type: 'number',
|
||||
label: 'Início (s)',
|
||||
admin: {
|
||||
description: 'Início do segmento quando conhecido; vazio na janela de texto.',
|
||||
},
|
||||
},
|
||||
{
|
||||
name: 'model',
|
||||
type: 'text',
|
||||
label: 'Modelo',
|
||||
required: true,
|
||||
admin: {
|
||||
description: 'Modelo de embedding que gerou o vetor (muda ⇒ reindexar).',
|
||||
},
|
||||
},
|
||||
{
|
||||
name: 'dimensions',
|
||||
type: 'number',
|
||||
label: 'Dimensões',
|
||||
required: true,
|
||||
},
|
||||
{
|
||||
name: 'contentHash',
|
||||
type: 'text',
|
||||
label: 'Hash do conteúdo',
|
||||
required: true,
|
||||
admin: {
|
||||
description: 'Hash do texto + modelo; decide o pulo na reindexação.',
|
||||
},
|
||||
},
|
||||
{
|
||||
name: 'vector',
|
||||
type: 'json',
|
||||
label: 'Vetor',
|
||||
required: true,
|
||||
admin: {
|
||||
readOnly: true,
|
||||
description: 'Embedding L2-normalizado da unidade.',
|
||||
},
|
||||
},
|
||||
],
|
||||
}
|
||||
|
|
@ -1,7 +1,13 @@
|
|||
'use client'
|
||||
|
||||
import { useCampaignListFilterNavigation } from '@/components/campaign/shared/useCampaignListFilterNavigation'
|
||||
import { ACERVO_SORT_OPTIONS, acervoSortIsDuration, type AcervoSortKey } from '@/lib/acervoListSort'
|
||||
import {
|
||||
ACERVO_SORT_OPTIONS,
|
||||
ACERVO_THEME_SORT_OPTIONS,
|
||||
acervoSortIsDuration,
|
||||
type AcervoThemeSortKey,
|
||||
} from '@/lib/acervoListSort'
|
||||
import { cn } from '@/lib/utils'
|
||||
import {
|
||||
buildRecordingListHref,
|
||||
type RecordingListState,
|
||||
|
|
@ -16,31 +22,72 @@ import { buildSpeechListHref, type SpeechListState } from '@/utilities/speech/sp
|
|||
* the boundary, so the client-safe domain contracts are imported inside this
|
||||
* component and the page passes only the serializable state and the honest
|
||||
* line shown while a duration order is active (it lists measured rows only).
|
||||
*
|
||||
* C229 — the web theme mode opens with "Mais relevantes" (the semantic default,
|
||||
* never serialized) and keeps "Mais recentes" as a real, serializable state;
|
||||
* recordings (and the exact mode) keep the C216 vocabulary untouched.
|
||||
*/
|
||||
type AcervoSortState = RecordingListState | SpeechListState
|
||||
|
||||
export const AcervoSortSelect = ({ state, hint }: { state: AcervoSortState; hint: string }) => {
|
||||
export const AcervoSortSelect = ({
|
||||
state,
|
||||
hint,
|
||||
themeUnavailable = false,
|
||||
}: {
|
||||
state: AcervoSortState
|
||||
hint: string
|
||||
/** C229 — the engine is down: the list is literal, so show the exact-mode order. */
|
||||
themeUnavailable?: boolean
|
||||
}) => {
|
||||
const { navigate, isPending } = useCampaignListFilterNavigation({
|
||||
state,
|
||||
toHref: (next) =>
|
||||
next.source === 'enviadas' ? buildRecordingListHref(next, 1) : buildSpeechListHref(next, 1),
|
||||
})
|
||||
|
||||
const change = (value: AcervoSortKey) => {
|
||||
navigate({ ...state, page: 1, sort: value === 'recentes' ? undefined : value })
|
||||
const isWebTheme = state.source === 'internet' && state.mode === 'tema'
|
||||
const isSpeechTheme = isWebTheme && !themeUnavailable
|
||||
const options = isSpeechTheme ? ACERVO_THEME_SORT_OPTIONS : ACERVO_SORT_OPTIONS
|
||||
const selected: AcervoThemeSortKey = state.sort ?? (isSpeechTheme ? 'relevancia' : 'recentes')
|
||||
|
||||
const change = (value: AcervoThemeSortKey) => {
|
||||
// Relevance means "no sort" in the theme mode; "recentes" does so in every
|
||||
// other path (each mode's never-serialized default).
|
||||
if (value === 'relevancia' || (value === 'recentes' && !isSpeechTheme)) {
|
||||
navigate({ ...state, page: 1, sort: undefined })
|
||||
return
|
||||
}
|
||||
navigate({ ...state, page: 1, sort: value })
|
||||
}
|
||||
|
||||
return (
|
||||
<div className="flex flex-col items-end gap-1">
|
||||
<label className="flex items-center gap-2 text-xs text-muted-foreground">
|
||||
<div
|
||||
className={cn(
|
||||
'flex flex-col items-end gap-1',
|
||||
// C229 — the web theme control takes the full row on mobile (design
|
||||
// scene 02): one bordered line with the label left and the order right.
|
||||
isWebTheme && 'w-full md:w-auto',
|
||||
)}
|
||||
>
|
||||
<label
|
||||
className={cn(
|
||||
'flex items-center gap-2 text-xs text-muted-foreground',
|
||||
isWebTheme &&
|
||||
'min-h-11 w-full justify-between rounded-lg border border-input bg-white px-3 md:min-h-0 md:w-auto md:justify-start md:gap-2 md:rounded-none md:border-0 md:bg-transparent md:px-0',
|
||||
)}
|
||||
>
|
||||
Ordenar por
|
||||
<select
|
||||
value={state.sort ?? 'recentes'}
|
||||
value={selected}
|
||||
disabled={isPending}
|
||||
className="min-h-10 rounded-md border border-input bg-white px-3 py-2 text-sm text-foreground disabled:opacity-70"
|
||||
onChange={(event) => change(event.target.value as AcervoSortKey)}
|
||||
className={cn(
|
||||
'min-h-10 rounded-md border border-input bg-white px-3 py-2 text-sm text-foreground disabled:opacity-70',
|
||||
isWebTheme &&
|
||||
'min-h-11 rounded-md border-0 bg-transparent px-0 text-sm font-medium outline-none md:min-h-10 md:rounded-md md:border md:border-input md:bg-white md:px-3',
|
||||
)}
|
||||
onChange={(event) => change(event.target.value as AcervoThemeSortKey)}
|
||||
>
|
||||
{ACERVO_SORT_OPTIONS.map((option) => (
|
||||
{options.map((option) => (
|
||||
<option key={option.value} value={option.value}>
|
||||
{option.label}
|
||||
</option>
|
||||
|
|
|
|||
|
|
@ -15,7 +15,8 @@ import { cn } from '@/lib/utils'
|
|||
* The standard campaign list empty state (icon, title, description, optional
|
||||
* CTA). Rendered inside `CampaignListResults` so it dims with the shared
|
||||
* transition like the rows it replaces. `className` / `mediaClassName` /
|
||||
* `contentClassName` let a caller tune the shell without forking it (C192).
|
||||
* `contentClassName` / `headerClassName` let a caller tune the shell without
|
||||
* forking it (C192/C229).
|
||||
*/
|
||||
export const CampaignListEmptyState = ({
|
||||
icon: Icon,
|
||||
|
|
@ -25,6 +26,7 @@ export const CampaignListEmptyState = ({
|
|||
className,
|
||||
mediaClassName,
|
||||
contentClassName,
|
||||
headerClassName,
|
||||
}: {
|
||||
icon: LucideIcon
|
||||
title: string
|
||||
|
|
@ -34,9 +36,11 @@ export const CampaignListEmptyState = ({
|
|||
className?: string
|
||||
mediaClassName?: string
|
||||
contentClassName?: string
|
||||
/** C229 — widens the copy of the theme empty state (design scene 03). */
|
||||
headerClassName?: string
|
||||
}) => (
|
||||
<Empty className={cn('min-h-72 border', className)}>
|
||||
<EmptyHeader>
|
||||
<EmptyHeader className={headerClassName}>
|
||||
<EmptyMedia variant="icon" className={mediaClassName}>
|
||||
<Icon aria-hidden="true" />
|
||||
</EmptyMedia>
|
||||
|
|
|
|||
|
|
@ -28,6 +28,7 @@ export const CampaignSearchModeControl = ({
|
|||
themeUnavailable = false,
|
||||
onSelect,
|
||||
relatedHint,
|
||||
unavailableHint = 'Indisponível agora; mostramos o termo exato.',
|
||||
}: {
|
||||
activeMode: SpeechSearchMode
|
||||
/** The theme expansion is down; the selector reflects the fallback. */
|
||||
|
|
@ -35,6 +36,12 @@ export const CampaignSearchModeControl = ({
|
|||
onSelect: (mode: SpeechSearchMode) => void
|
||||
/** Desktop hint of the corpus ("Encontra falas relacionadas pelo sentido."). */
|
||||
relatedHint: string
|
||||
/**
|
||||
* C229 — the degraded helper. The speech sources use the approved copy of the
|
||||
* fallback scene, which stays visible on mobile too (the success hint does
|
||||
* not); the recordings keep their own line.
|
||||
*/
|
||||
unavailableHint?: string
|
||||
}) => (
|
||||
<div className="mt-3 flex flex-col gap-3 border-t border-border pt-3 md:flex-row md:flex-wrap md:items-center md:justify-between">
|
||||
<div className="flex w-full items-center gap-2 md:w-auto">
|
||||
|
|
@ -76,8 +83,13 @@ export const CampaignSearchModeControl = ({
|
|||
</button>
|
||||
</div>
|
||||
</div>
|
||||
<p className="hidden text-xs text-muted-foreground md:block">
|
||||
{themeUnavailable ? 'Indisponível agora; mostramos o termo exato.' : relatedHint}
|
||||
<p
|
||||
className={cn(
|
||||
'text-xs text-muted-foreground',
|
||||
themeUnavailable ? 'block' : 'hidden md:block',
|
||||
)}
|
||||
>
|
||||
{themeUnavailable ? unavailableHint : relatedHint}
|
||||
</p>
|
||||
</div>
|
||||
)
|
||||
|
|
|
|||
|
|
@ -269,6 +269,7 @@ export const SpeechAcervoFilters = ({
|
|||
themeUnavailable={themeUnavailable}
|
||||
onSelect={runMode}
|
||||
relatedHint="Encontra falas relacionadas pelo sentido."
|
||||
unavailableHint="Você pode tentar “Por tema” novamente."
|
||||
/>
|
||||
|
||||
<div
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
import { ExternalLinkIcon, PlayIcon, SparklesIcon } from 'lucide-react'
|
||||
import { ExternalLinkIcon, PlayIcon } from 'lucide-react'
|
||||
import Link from 'next/link'
|
||||
import type { ReactNode } from 'react'
|
||||
|
||||
|
|
@ -9,8 +9,12 @@ import {
|
|||
type SpeechChipGroup,
|
||||
} from '@/components/campaign/speech/SpeechResultChips'
|
||||
import { SpeechResultThumbnail } from '@/components/campaign/speech/SpeechResultThumbnail'
|
||||
import {
|
||||
SpeechLiteralMatchBadge,
|
||||
SpeechSemanticBadges,
|
||||
SpeechSemanticProvenance,
|
||||
} from '@/components/campaign/speech/SpeechSemanticEvidence'
|
||||
import { Button } from '@/components/ui/button'
|
||||
import type { SpeechHighlightedExcerpt } from '@/lib/speechHighlight'
|
||||
import type { SpeechListItemViewModel } from '@/utilities/speech/speechViewModels'
|
||||
|
||||
/**
|
||||
|
|
@ -23,42 +27,6 @@ const MAX_KEYWORD_CHIPS = 3
|
|||
|
||||
const MetaSeparator = () => <span aria-hidden="true">·</span>
|
||||
|
||||
/**
|
||||
* C192 — the theme result provenance: the real excerpt that matched the theme,
|
||||
* with the expanded term highlighted. It replaces the common excerpt (never
|
||||
* coexists with it) and carries no score.
|
||||
*/
|
||||
const SpeechThemeProvenance = ({ excerpt }: { excerpt: SpeechHighlightedExcerpt }) => {
|
||||
if (excerpt.parts.length === 0) return null
|
||||
|
||||
return (
|
||||
<div className="rounded-lg border-l-2 border-primary bg-primary/[0.035] px-3 py-2.5">
|
||||
<p className="text-[11px] font-semibold uppercase tracking-wide text-primary">
|
||||
Por que apareceu
|
||||
</p>
|
||||
<SpeechExcerpt
|
||||
excerpt={excerpt}
|
||||
className="mt-1 text-sm leading-6 text-foreground/90"
|
||||
quoted
|
||||
/>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
const SpeechThemeBadges = ({ matchedTextSearch }: { matchedTextSearch: boolean }) => (
|
||||
<div className="flex flex-wrap items-center gap-1.5">
|
||||
<span className="inline-flex h-5 items-center gap-1 rounded-full border border-primary/20 bg-primary/10 px-2 py-0.5 text-xs font-semibold text-primary">
|
||||
<SparklesIcon className="size-3" aria-hidden="true" />
|
||||
Tema
|
||||
</span>
|
||||
{matchedTextSearch ? (
|
||||
<span className="inline-flex h-5 items-center rounded-full border border-border bg-background px-2 py-0.5 text-xs font-medium text-foreground">
|
||||
Termo exato
|
||||
</span>
|
||||
) : null}
|
||||
</div>
|
||||
)
|
||||
|
||||
const ActionLink = ({
|
||||
href,
|
||||
children,
|
||||
|
|
@ -109,7 +77,7 @@ const SpeechMetaLine = ({
|
|||
|
||||
/**
|
||||
* The primary CTA of every result card: a link to the speech (or its trecho)
|
||||
* whose variant/width the caller owns (C192 adds the theme appearance).
|
||||
* whose variant/width the caller owns (C229 adds the theme appearance).
|
||||
*/
|
||||
const SpeechWatchButton = ({
|
||||
speech,
|
||||
|
|
@ -169,13 +137,18 @@ export const SpeechResultCard = ({
|
|||
}) => {
|
||||
// C174 (option B): a speech that surfaced only because a cut matched the term
|
||||
// shows the origin note instead of an excerpt the speech does not contain, and
|
||||
// the actions sit at the top right (approved scene 05). C192 — a theme result
|
||||
// outranks this branch: the theme evidence lives in the speech itself.
|
||||
if (!speech.matchedTextSearch && speech.themeMatchTerm === null) {
|
||||
// the actions sit at the top right (approved scene 05). C229 — a semantic
|
||||
// result outranks this branch: the evidence lives in the speech itself.
|
||||
if (!speech.matchedTextSearch && !speech.semanticMatch) {
|
||||
return (
|
||||
<article className="rounded-xl border bg-card p-4">
|
||||
<div className="flex flex-col gap-3 md:flex-row md:items-start md:justify-between md:gap-4">
|
||||
<div className="min-w-0">
|
||||
{speech.literalFallback ? (
|
||||
<div className="mb-2 flex flex-wrap items-center gap-1.5">
|
||||
<SpeechLiteralMatchBadge />
|
||||
</div>
|
||||
) : null}
|
||||
<SpeechMetaLine speech={speech} originLabel />
|
||||
<p className="mt-2 text-sm text-muted-foreground">
|
||||
A fala não contém o termo exato; ela aparece porque um corte vinculado corresponde à
|
||||
|
|
@ -212,15 +185,15 @@ export const SpeechResultCard = ({
|
|||
className: 'font-normal text-muted-foreground',
|
||||
},
|
||||
]
|
||||
const isTheme = speech.themeMatchTerm !== null
|
||||
// C192 — a theme card only offers the trecho seek when the speech really
|
||||
const isTheme = speech.semanticMatch
|
||||
// C229 — a semantic card only offers the trecho seek when the speech really
|
||||
// contains the literal query (a stopword-only segment hit must not claim it).
|
||||
const watchLabel =
|
||||
speech.matchKind === 'segment' && speech.matchedTextSearch ? 'Assistir no trecho' : 'Ver fala'
|
||||
// C192 — the theme card replaces the common excerpt with the provenance block;
|
||||
// the two never coexist (design gate).
|
||||
// C229 — the semantic card replaces the common excerpt with the evidence
|
||||
// block; the two never coexist (design gate).
|
||||
const bodyNode = isTheme ? (
|
||||
<SpeechThemeProvenance excerpt={speech.excerpt} />
|
||||
<SpeechSemanticProvenance excerpt={speech.excerpt} />
|
||||
) : (
|
||||
<SpeechExcerpt
|
||||
excerpt={speech.excerpt}
|
||||
|
|
@ -234,11 +207,18 @@ export const SpeechResultCard = ({
|
|||
<article className="rounded-xl border bg-card p-4">
|
||||
{isTheme ? (
|
||||
<div className="flex flex-col items-start gap-2">
|
||||
<SpeechThemeBadges matchedTextSearch={speech.matchedTextSearch} />
|
||||
<SpeechSemanticBadges matchedTextSearch={speech.matchedTextSearch} />
|
||||
<SpeechMetaLine speech={speech} />
|
||||
</div>
|
||||
) : (
|
||||
<SpeechMetaLine speech={speech} />
|
||||
<div className="flex flex-col items-start gap-2">
|
||||
{speech.literalFallback ? (
|
||||
<div className="flex flex-wrap items-center gap-1.5">
|
||||
<SpeechLiteralMatchBadge />
|
||||
</div>
|
||||
) : null}
|
||||
<SpeechMetaLine speech={speech} />
|
||||
</div>
|
||||
)}
|
||||
|
||||
<div className={isTheme ? 'mt-3' : 'mt-2'}>
|
||||
|
|
|
|||
58
src/components/campaign/speech/SpeechSemanticEvidence.tsx
Normal file
58
src/components/campaign/speech/SpeechSemanticEvidence.tsx
Normal file
|
|
@ -0,0 +1,58 @@
|
|||
import { SparklesIcon } from 'lucide-react'
|
||||
|
||||
import { SpeechExcerpt } from '@/components/campaign/speech/SpeechExcerpt'
|
||||
import type { SpeechHighlightedExcerpt } from '@/lib/speechHighlight'
|
||||
|
||||
/**
|
||||
* C229 — the shared semantic marks of a result card (design scenes 01/02): the
|
||||
* "Tema" seal (the engine surfaced the row by meaning), the coexisting "Termo
|
||||
* exato" seal (the row also contains the literal query) and the evidence block
|
||||
* "Trecho mais próximo do tema" — a real passage, no highlight and no score.
|
||||
* Both the Câmara and the web cards render exactly this structure.
|
||||
*/
|
||||
|
||||
export const SpeechSemanticBadges = ({ matchedTextSearch }: { matchedTextSearch: boolean }) => (
|
||||
<div className="flex flex-wrap items-center gap-1.5">
|
||||
<span className="inline-flex h-5 items-center gap-1 rounded-full border border-primary/20 bg-primary/10 px-2 py-0.5 text-xs font-semibold text-primary">
|
||||
<SparklesIcon className="size-3" aria-hidden="true" />
|
||||
Tema
|
||||
</span>
|
||||
{matchedTextSearch ? (
|
||||
<span className="inline-flex h-5 items-center rounded-full border border-border bg-background px-2 py-0.5 text-xs font-medium text-foreground">
|
||||
Termo exato
|
||||
</span>
|
||||
) : null}
|
||||
</div>
|
||||
)
|
||||
|
||||
/**
|
||||
* C229 — the seal of a literal result inside the degraded theme path (design
|
||||
* scene 04): the engine was down, the list below is by term. The plain exact
|
||||
* search never renders it (the page only passes `literalFallback` when the
|
||||
* actor asked for the theme).
|
||||
*/
|
||||
export const SpeechLiteralMatchBadge = () => (
|
||||
<span
|
||||
data-testid="speech-literal-badge"
|
||||
className="inline-flex h-5 items-center rounded-full border border-border bg-background px-2 py-0.5 text-xs font-medium text-foreground"
|
||||
>
|
||||
Termo exato
|
||||
</span>
|
||||
)
|
||||
|
||||
export const SpeechSemanticProvenance = ({ excerpt }: { excerpt: SpeechHighlightedExcerpt }) => {
|
||||
if (excerpt.parts.length === 0) return null
|
||||
|
||||
return (
|
||||
<div className="rounded-lg border-l-2 border-primary bg-primary/[0.035] px-3 py-2.5">
|
||||
<p className="text-[11px] font-semibold uppercase tracking-wide text-primary">
|
||||
Trecho mais próximo do tema
|
||||
</p>
|
||||
<SpeechExcerpt
|
||||
excerpt={excerpt}
|
||||
className="mt-1 text-sm leading-6 text-foreground/90"
|
||||
quoted
|
||||
/>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
|
@ -3,9 +3,10 @@ import { TriangleAlertIcon } from 'lucide-react'
|
|||
import { Alert, AlertDescription, AlertTitle } from '@/components/ui/Alert'
|
||||
|
||||
/**
|
||||
* C192 — the discreet fallback notice when `mode=tema` was asked but the theme
|
||||
* expansion was unavailable. The list below shows the literal results; the
|
||||
* notice keeps the user's request honest instead of hiding the downgrade.
|
||||
* C229 — the discreet fallback notice when `mode=tema` was asked but the sense
|
||||
* engine was unavailable (no provider or an empty index). The list below shows
|
||||
* the literal results; the notice keeps the user's request honest instead of
|
||||
* hiding the downgrade.
|
||||
*/
|
||||
export const SpeechThemeFallbackNotice = () => (
|
||||
<Alert
|
||||
|
|
@ -15,11 +16,10 @@ export const SpeechThemeFallbackNotice = () => (
|
|||
>
|
||||
<TriangleAlertIcon aria-hidden="true" />
|
||||
<AlertTitle className="text-sm font-medium">
|
||||
A busca por tema está indisponível agora.
|
||||
A busca por sentido está indisponível agora.
|
||||
</AlertTitle>
|
||||
<AlertDescription className="text-xs leading-5">
|
||||
Você pediu “Por tema”. Para não interromper o trabalho, mostramos abaixo os resultados por
|
||||
termo exato.
|
||||
Para não interromper o trabalho, mostramos resultados por correspondência de termos.
|
||||
</AlertDescription>
|
||||
</Alert>
|
||||
)
|
||||
|
|
|
|||
|
|
@ -7,6 +7,11 @@ import {
|
|||
type SpeechChipGroup,
|
||||
} from '@/components/campaign/speech/SpeechResultChips'
|
||||
import { SpeechResultThumbnail } from '@/components/campaign/speech/SpeechResultThumbnail'
|
||||
import {
|
||||
SpeechLiteralMatchBadge,
|
||||
SpeechSemanticBadges,
|
||||
SpeechSemanticProvenance,
|
||||
} from '@/components/campaign/speech/SpeechSemanticEvidence'
|
||||
import { WebSpeechPlatformPill } from '@/components/campaign/speech/WebSpeechPlatformPill'
|
||||
import { Button } from '@/components/ui/button'
|
||||
import type { WebSpeechListItemViewModel } from '@/utilities/speech/speechViewModels'
|
||||
|
|
@ -19,7 +24,9 @@ const MAX_SCOPE_CHIPS = 2
|
|||
* C216 — one web speech in the "Falas na internet" list (design scenes 01/03/05):
|
||||
* the cover (or the honest neutral placeholder), the platform pill on the cover
|
||||
* and beside the date, the matching excerpt with the search highlight and the
|
||||
* single "Ver fala" action.
|
||||
* single "Ver fala" action. C229 — in the theme mode the card carries the
|
||||
* "Tema"/"Termo exato" seals and the real evidence block ("Trecho mais próximo
|
||||
* do tema") instead of the highlighted excerpt.
|
||||
*/
|
||||
export const WebSpeechResultCard = ({ speech }: { speech: WebSpeechListItemViewModel }) => {
|
||||
const { excerpt, thumbnailUrl, watchHref } = speech
|
||||
|
|
@ -29,23 +36,24 @@ export const WebSpeechResultCard = ({ speech }: { speech: WebSpeechListItemViewM
|
|||
{ key: 'scopes', items: speech.scopes, max: MAX_SCOPE_CHIPS },
|
||||
]
|
||||
const hasChips = Boolean(speech.topics.length || speech.scopes.length)
|
||||
const isTheme = speech.semanticMatch
|
||||
|
||||
return (
|
||||
<article className="grid gap-4 rounded-xl border border-border p-4 md:grid-cols-[152px_minmax(0,1fr)_auto]">
|
||||
<article className="grid gap-4 rounded-xl border border-border p-4 md:grid-cols-[128px_minmax(0,1fr)_auto]">
|
||||
<div className="relative">
|
||||
{thumbnailUrl ? (
|
||||
<SpeechResultThumbnail
|
||||
href={watchHref}
|
||||
src={thumbnailUrl}
|
||||
label={`Ver fala: ${speech.title}`}
|
||||
className="aspect-video w-full md:w-[152px]"
|
||||
fullWidthOnMobile
|
||||
/>
|
||||
) : (
|
||||
<Link
|
||||
href={watchHref}
|
||||
aria-hidden="true"
|
||||
tabIndex={-1}
|
||||
className="relative grid aspect-video w-full place-items-center overflow-hidden rounded-lg bg-stone-200 text-stone-500 md:w-[152px]"
|
||||
className="relative grid h-28 w-full place-items-center overflow-hidden rounded-lg bg-stone-200 text-stone-500 md:h-20 md:w-32"
|
||||
>
|
||||
<VideoIcon className="size-7" aria-hidden="true" />
|
||||
<span className="absolute bottom-2 text-[10px] font-semibold">SEM CAPA</span>
|
||||
|
|
@ -62,13 +70,23 @@ export const WebSpeechResultCard = ({ speech }: { speech: WebSpeechListItemViewM
|
|||
</div>
|
||||
|
||||
<div className="min-w-0">
|
||||
<div className="flex flex-wrap items-center gap-2">
|
||||
{isTheme ? <SpeechSemanticBadges matchedTextSearch={speech.matchedTextSearch} /> : null}
|
||||
{!isTheme && speech.literalFallback ? <SpeechLiteralMatchBadge /> : null}
|
||||
<div
|
||||
className={`flex flex-wrap items-center gap-2 ${isTheme || speech.literalFallback ? 'mt-2' : ''}`}
|
||||
>
|
||||
<WebSpeechPlatformPill platform={speech.platform.value} label={speech.platform.label} />
|
||||
<span className="text-xs text-muted-foreground">{metaLine}</span>
|
||||
</div>
|
||||
<h3 className="mt-2 text-sm font-semibold">{speech.title}</h3>
|
||||
|
||||
<SpeechExcerpt excerpt={excerpt} className="mt-2 text-sm leading-6 text-foreground/90" />
|
||||
{isTheme ? (
|
||||
<div className="mt-2">
|
||||
<SpeechSemanticProvenance excerpt={excerpt} />
|
||||
</div>
|
||||
) : (
|
||||
<SpeechExcerpt excerpt={excerpt} className="mt-2 text-sm leading-6 text-foreground/90" />
|
||||
)}
|
||||
|
||||
{hasChips ? (
|
||||
<div className="mt-2 flex flex-wrap gap-1.5">
|
||||
|
|
|
|||
|
|
@ -20,7 +20,9 @@ const ACERVO_SORT_SET = new Set<string>(ACERVO_SORT_OPTIONS.map(({ value }) => v
|
|||
const firstParam = (value: string | string[] | undefined): string | undefined =>
|
||||
Array.isArray(value) ? value[0] : value
|
||||
|
||||
/** Only the non-default orders are meaningful; anything else (and `recentes`) is the default. */
|
||||
/**
|
||||
* Only the non-default orders are meaningful; anything else (and `recentes`) is the default.
|
||||
*/
|
||||
export const parseAcervoSort = (raw: string | string[] | undefined): AcervoSortKey | undefined => {
|
||||
const value = firstParam(raw)
|
||||
return value && value !== 'recentes' && ACERVO_SORT_SET.has(value)
|
||||
|
|
@ -28,6 +30,34 @@ export const parseAcervoSort = (raw: string | string[] | undefined): AcervoSortK
|
|||
: undefined
|
||||
}
|
||||
|
||||
/**
|
||||
* C229 — the ordering vocabulary of the theme mode of the speech acervo:
|
||||
* relevance is the default (never serialized) and the explicit date/duration
|
||||
* order must stay askable. Recordings and the public Central keep
|
||||
* `ACERVO_SORT_OPTIONS` — their theme mode is not the semantic engine.
|
||||
*/
|
||||
export const ACERVO_THEME_SORT_OPTIONS = [
|
||||
{ value: 'relevancia', label: 'Mais relevantes' },
|
||||
...ACERVO_SORT_OPTIONS,
|
||||
] as const
|
||||
|
||||
export type AcervoThemeSortKey = (typeof ACERVO_THEME_SORT_OPTIONS)[number]['value']
|
||||
|
||||
const ACERVO_THEME_SORT_SET = new Set<string>(ACERVO_THEME_SORT_OPTIONS.map(({ value }) => value))
|
||||
|
||||
/**
|
||||
* Theme-aware parse: `relevancia` (and anything unknown) parses away as the
|
||||
* default, while `recentes` is a REAL state here — the exact-mode default has
|
||||
* to stay explicitly askable after relevance became the theme default.
|
||||
*/
|
||||
export const parseAcervoThemeSort = (
|
||||
raw: string | string[] | undefined,
|
||||
): AcervoSortKey | undefined => {
|
||||
const value = firstParam(raw)
|
||||
if (!value || value === 'relevancia' || !ACERVO_THEME_SORT_SET.has(value)) return undefined
|
||||
return value as AcervoSortKey
|
||||
}
|
||||
|
||||
/** Duration orders only make sense over rows with a measured duration. */
|
||||
export const acervoSortIsDuration = (sort: AcervoSortKey | undefined): boolean =>
|
||||
sort === 'duracao_maior' || sort === 'duracao_menor'
|
||||
|
|
|
|||
|
|
@ -61,7 +61,7 @@ const buildNormalizedIndex = (text: string): NormalizedIndex => {
|
|||
}
|
||||
|
||||
/** Normalized search terms (words) of the query, in order, duplicates kept out. */
|
||||
export const speechSearchTerms = (query: string): string[] => [
|
||||
const speechSearchTerms = (query: string): string[] => [
|
||||
...new Set(
|
||||
normalizeForSearch(query)
|
||||
.split(' ')
|
||||
|
|
|
|||
297
src/lib/speechSemantic.ts
Normal file
297
src/lib/speechSemantic.ts
Normal file
|
|
@ -0,0 +1,297 @@
|
|||
/**
|
||||
* C229 — pure semantic engine of the speech acervo. The index stores
|
||||
* L2-normalized vectors per speech and per retrieval unit (ASR segment or text
|
||||
* window); this module owns the vector math, the window splitter used when a
|
||||
* speech has no segments, the deterministic content hash of the index and the
|
||||
* ranking rules (threshold, relevance default, explicit order override). No IO,
|
||||
* Payload or Next: the loader and the index CLI share it and the unit tests
|
||||
* pin it without a network.
|
||||
*/
|
||||
|
||||
/** The provisional cut-off of the theme mode — calibrated by `acervo:index --probe`. */
|
||||
export const SPEECH_SEMANTIC_MIN_COSINE = 0.4
|
||||
|
||||
/** Text windows of speeches without ASR segments: ~1200 chars, ~200 of overlap. */
|
||||
const SPEECH_WINDOW_SIZE_CHARS = 1200
|
||||
const SPEECH_WINDOW_OVERLAP_CHARS = 200
|
||||
|
||||
export type SpeechSemanticVector = readonly number[]
|
||||
|
||||
export const dotProduct = (a: SpeechSemanticVector, b: SpeechSemanticVector): number => {
|
||||
if (a.length !== b.length) return 0
|
||||
let sum = 0
|
||||
for (let index = 0; index < a.length; index += 1) sum += a[index] * b[index]
|
||||
return sum
|
||||
}
|
||||
|
||||
const vectorNorm = (vector: SpeechSemanticVector): number => Math.sqrt(dotProduct(vector, vector))
|
||||
|
||||
/** L2-normalizes; a zero/degenerate vector normalizes to zeros (never NaN). */
|
||||
export const normalizeVector = (vector: SpeechSemanticVector): number[] => {
|
||||
const norm = vectorNorm(vector)
|
||||
if (norm === 0) return vector.map(() => 0)
|
||||
return vector.map((value) => value / norm)
|
||||
}
|
||||
|
||||
/**
|
||||
* Cosine of two vectors. The index normalizes on write, but this stays a real
|
||||
* cosine so a non-normalized fixture never turns into a silent dot product.
|
||||
*/
|
||||
export const cosineSimilarity = (a: SpeechSemanticVector, b: SpeechSemanticVector): number => {
|
||||
if (a.length !== b.length || a.length === 0) return 0
|
||||
const normA = vectorNorm(a)
|
||||
const normB = vectorNorm(b)
|
||||
if (normA === 0 || normB === 0) return 0
|
||||
return dotProduct(a, b) / (normA * normB)
|
||||
}
|
||||
|
||||
/** The speech-level vector: element-wise mean of its unit vectors, normalized. */
|
||||
export const meanPoolVectors = (vectors: readonly SpeechSemanticVector[]): number[] => {
|
||||
const usable = vectors.filter((vector) => vector.length > 0)
|
||||
if (usable.length === 0) return []
|
||||
const dimensions = usable[0].length
|
||||
const sums = new Array<number>(dimensions).fill(0)
|
||||
let counted = 0
|
||||
for (const vector of usable) {
|
||||
if (vector.length !== dimensions) continue
|
||||
for (let index = 0; index < dimensions; index += 1) sums[index] += vector[index]
|
||||
counted += 1
|
||||
}
|
||||
if (counted === 0) return []
|
||||
return normalizeVector(sums.map((sum) => sum / counted))
|
||||
}
|
||||
|
||||
export type SpeechTextWindow = {
|
||||
order: number
|
||||
start: number
|
||||
end: number
|
||||
text: string
|
||||
}
|
||||
|
||||
/**
|
||||
* Splits a text into overlapping windows without cutting a word in half: the
|
||||
* end retreats to the last whitespace inside the window when possible. The
|
||||
* SAME function builds the index and re-reads the evidence window, so `order`
|
||||
* is stable.
|
||||
*/
|
||||
export const speechTextWindows = (
|
||||
text: string,
|
||||
{
|
||||
size = SPEECH_WINDOW_SIZE_CHARS,
|
||||
overlap = SPEECH_WINDOW_OVERLAP_CHARS,
|
||||
}: { size?: number; overlap?: number } = {},
|
||||
): SpeechTextWindow[] => {
|
||||
if (!Number.isInteger(size) || size <= 0) return []
|
||||
const safeOverlap = Math.min(Math.max(overlap, 0), size - 1)
|
||||
const windows: SpeechTextWindow[] = []
|
||||
let start = 0
|
||||
while (start < text.length) {
|
||||
let end = Math.min(start + size, text.length)
|
||||
if (end < text.length) {
|
||||
const pivot = text.lastIndexOf(' ', end)
|
||||
if (pivot > start) end = pivot
|
||||
}
|
||||
const chunk = text.slice(start, end).trim()
|
||||
if (chunk) windows.push({ order: windows.length, start, end, text: chunk })
|
||||
if (end >= text.length) break
|
||||
start = Math.max(end - safeOverlap, start + 1)
|
||||
}
|
||||
return windows
|
||||
}
|
||||
|
||||
/**
|
||||
* FNV-1a over `model\0text` — the staleness key of one index row. Cheap and
|
||||
* deterministic; it only decides whether a re-import must re-embed.
|
||||
*/
|
||||
export const speechEmbeddingHash = (text: string, model: string): string => {
|
||||
const source = `${model}\u0000${text}`
|
||||
let hash = 0x811c9dc5
|
||||
for (let index = 0; index < source.length; index += 1) {
|
||||
hash ^= source.charCodeAt(index)
|
||||
hash = Math.imul(hash, 0x01000193)
|
||||
}
|
||||
return (hash >>> 0).toString(16).padStart(8, '0')
|
||||
}
|
||||
|
||||
export type SpeechSemanticCandidate = {
|
||||
id: number
|
||||
/** Câmara wall-clock string; the only tiebreak of the relevance order. */
|
||||
speechAt: string
|
||||
durationSeconds?: number | null
|
||||
vector: SpeechSemanticVector | null
|
||||
}
|
||||
|
||||
export type SpeechSemanticHit = {
|
||||
id: number
|
||||
speechAt: string
|
||||
durationSeconds: number | null
|
||||
score: number
|
||||
}
|
||||
|
||||
/** Newest first; id breaks exact ties so the order is total and stable. */
|
||||
const byDateDesc = (a: SpeechSemanticHit, b: SpeechSemanticHit): number =>
|
||||
a.speechAt === b.speechAt ? a.id - b.id : a.speechAt < b.speechAt ? 1 : -1
|
||||
|
||||
/**
|
||||
* Similarity ranking with the honest cut-off: candidates without a usable
|
||||
* vector (wrong dimension included) and hits below `minScore` are dropped, the
|
||||
* rest sorts by score desc with the date (and then the id) as tiebreak.
|
||||
*/
|
||||
export const rankSemanticHits = (
|
||||
query: SpeechSemanticVector,
|
||||
candidates: readonly SpeechSemanticCandidate[],
|
||||
{ minScore = SPEECH_SEMANTIC_MIN_COSINE }: { minScore?: number } = {},
|
||||
): SpeechSemanticHit[] => {
|
||||
const hits: SpeechSemanticHit[] = []
|
||||
for (const candidate of candidates) {
|
||||
if (!candidate.vector || candidate.vector.length !== query.length) continue
|
||||
const score = cosineSimilarity(query, candidate.vector)
|
||||
if (score < minScore) continue
|
||||
hits.push({
|
||||
id: candidate.id,
|
||||
speechAt: candidate.speechAt,
|
||||
durationSeconds: candidate.durationSeconds ?? null,
|
||||
score,
|
||||
})
|
||||
}
|
||||
return hits.sort((a, b) => b.score - a.score || byDateDesc(a, b))
|
||||
}
|
||||
|
||||
export type SpeechSemanticSort = 'relevancia' | 'recentes' | 'duracao_maior' | 'duracao_menor'
|
||||
|
||||
/**
|
||||
* The explicit user order overrides relevance without re-filtering: the hits
|
||||
* already passed the threshold, and `Array.sort` is stable, so equal dates or
|
||||
* durations keep the relevance order.
|
||||
*/
|
||||
export const sortSemanticHits = (
|
||||
hits: readonly SpeechSemanticHit[],
|
||||
sort: SpeechSemanticSort,
|
||||
): SpeechSemanticHit[] => {
|
||||
const sorted = [...hits]
|
||||
switch (sort) {
|
||||
case 'recentes':
|
||||
return sorted.sort(byDateDesc)
|
||||
case 'duracao_maior':
|
||||
return sorted.sort(
|
||||
(a, b) => (b.durationSeconds ?? -1) - (a.durationSeconds ?? -1) || byDateDesc(a, b),
|
||||
)
|
||||
case 'duracao_menor':
|
||||
return sorted.sort(
|
||||
(a, b) =>
|
||||
(a.durationSeconds ?? Number.POSITIVE_INFINITY) -
|
||||
(b.durationSeconds ?? Number.POSITIVE_INFINITY) || byDateDesc(a, b),
|
||||
)
|
||||
default:
|
||||
return sorted
|
||||
}
|
||||
}
|
||||
|
||||
export type SpeechSemanticChunk = {
|
||||
/** Stable identity of the row inside its speech (the embedding row id). */
|
||||
key: number
|
||||
vector: SpeechSemanticVector | null
|
||||
}
|
||||
|
||||
/**
|
||||
* The evidence unit of one speech: the chunk closest to the query. `null` when
|
||||
* the speech has no usable chunk (the caller falls back to the lexical
|
||||
* excerpt). Ties resolve to the lowest key so the card never jumps.
|
||||
*/
|
||||
export const pickBestSemanticChunk = (
|
||||
query: SpeechSemanticVector,
|
||||
chunks: readonly SpeechSemanticChunk[],
|
||||
): { key: number; score: number } | null => {
|
||||
let best: { key: number; score: number } | null = null
|
||||
for (const chunk of chunks) {
|
||||
if (!chunk.vector || chunk.vector.length !== query.length) continue
|
||||
const score = cosineSimilarity(query, chunk.vector)
|
||||
if (!best || score > best.score || (score === best.score && chunk.key < best.key)) {
|
||||
best = { key: chunk.key, score }
|
||||
}
|
||||
}
|
||||
return best
|
||||
}
|
||||
|
||||
/**
|
||||
* The raw text the index and the evidence reader share for speeches without
|
||||
* ASR segments: the official transcript first (the real spelling), then the
|
||||
* summary, then the normalized search text as the last honest resort.
|
||||
*/
|
||||
export const speechSemanticSourceText = (speech: {
|
||||
officialTranscript?: string | null
|
||||
summary?: string | null
|
||||
searchText?: string | null
|
||||
}): string | null => {
|
||||
for (const candidate of [speech.officialTranscript, speech.summary, speech.searchText]) {
|
||||
if (typeof candidate === 'string' && candidate.trim() !== '') return candidate
|
||||
}
|
||||
return null
|
||||
}
|
||||
|
||||
type SpeechIndexUnit = {
|
||||
kind: 'segment' | 'window'
|
||||
/** Segment ordinal (the collection's) or the window index. */
|
||||
order: number
|
||||
startSeconds: number | null
|
||||
text: string
|
||||
}
|
||||
|
||||
export type SpeechIndexPlan = {
|
||||
units: SpeechIndexUnit[]
|
||||
/** Hash of every unit text + model: the CLI's skip key. */
|
||||
speechHash: string
|
||||
}
|
||||
|
||||
export type SpeechIndexSegment = {
|
||||
order: number
|
||||
startSeconds: number
|
||||
text: string
|
||||
}
|
||||
|
||||
/**
|
||||
* The retrieval units of one speech: the ASR segments when they exist, else
|
||||
* fixed text windows of the source text. `null` when the speech has no text at
|
||||
* all (the caller reports it as skipped). Pure: the index CLI and its tests
|
||||
* share it, and the evidence reader re-derives windows with the same splitter.
|
||||
*/
|
||||
export const planSpeechIndex = (
|
||||
speech: {
|
||||
officialTranscript?: string | null
|
||||
summary?: string | null
|
||||
searchText?: string | null
|
||||
segments?: readonly SpeechIndexSegment[]
|
||||
},
|
||||
{ model }: { model: string },
|
||||
): SpeechIndexPlan | null => {
|
||||
const units: SpeechIndexUnit[] = []
|
||||
const segments = speech.segments ?? []
|
||||
|
||||
if (segments.length > 0) {
|
||||
for (const segment of segments) {
|
||||
const text = segment.text.trim()
|
||||
if (!text) continue
|
||||
units.push({
|
||||
kind: 'segment',
|
||||
order: segment.order,
|
||||
startSeconds: segment.startSeconds,
|
||||
text,
|
||||
})
|
||||
}
|
||||
} else {
|
||||
const sourceText = speechSemanticSourceText(speech)
|
||||
if (sourceText) {
|
||||
for (const window of speechTextWindows(sourceText)) {
|
||||
units.push({ kind: 'window', order: window.order, startSeconds: null, text: window.text })
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (units.length === 0) return null
|
||||
|
||||
const speechHash = speechEmbeddingHash(
|
||||
units.map((unit) => `${unit.kind}:${unit.order}:${unit.text}`).join('\n'),
|
||||
model,
|
||||
)
|
||||
return { units, speechHash }
|
||||
}
|
||||
19452
src/migrations/20260926_034118_add_speech_embedding.json
Normal file
19452
src/migrations/20260926_034118_add_speech_embedding.json
Normal file
File diff suppressed because it is too large
Load diff
50
src/migrations/20260926_034118_add_speech_embedding.ts
Normal file
50
src/migrations/20260926_034118_add_speech_embedding.ts
Normal file
|
|
@ -0,0 +1,50 @@
|
|||
import { MigrateDownArgs, MigrateUpArgs, sql } from '@payloadcms/db-postgres'
|
||||
|
||||
/**
|
||||
* C229 — semantic index of the speech acervo (`speechEmbedding`).
|
||||
*
|
||||
* Hand-reduced from the generated diff: the two 2026-09-24 snapshots
|
||||
* (`add_content_piece_link_failure_reason`, `add_content_piece_people`) lost
|
||||
* the objects of the C215/C219 migrations, so the generator tried to recreate
|
||||
* `internet_speech_media`, the recording facet tables and the speech origin
|
||||
* columns — all of which already exist wherever the chain applied. The snapshot
|
||||
* written next to this file is the current full config state, so future
|
||||
* migrations diff from a healed baseline; this SQL stays scoped to the new
|
||||
* table.
|
||||
*/
|
||||
export async function up({ db }: MigrateUpArgs): Promise<void> {
|
||||
await db.execute(sql`
|
||||
CREATE TYPE "public"."enum_speech_embedding_kind" AS ENUM('speech', 'segment', 'window');
|
||||
CREATE TABLE "speech_embedding" (
|
||||
"id" serial PRIMARY KEY NOT NULL,
|
||||
"speech_id" integer NOT NULL,
|
||||
"kind" "enum_speech_embedding_kind" NOT NULL,
|
||||
"order" numeric,
|
||||
"start_seconds" numeric,
|
||||
"model" varchar NOT NULL,
|
||||
"dimensions" numeric NOT NULL,
|
||||
"content_hash" varchar NOT NULL,
|
||||
"vector" jsonb NOT NULL,
|
||||
"updated_at" timestamp(3) with time zone DEFAULT now() NOT NULL,
|
||||
"created_at" timestamp(3) with time zone DEFAULT now() NOT NULL
|
||||
);
|
||||
|
||||
ALTER TABLE "payload_locked_documents_rels" ADD COLUMN "speech_embedding_id" integer;
|
||||
ALTER TABLE "speech_embedding" ADD CONSTRAINT "speech_embedding_speech_id_speech_id_fk" FOREIGN KEY ("speech_id") REFERENCES "public"."speech"("id") ON DELETE set null ON UPDATE no action;
|
||||
CREATE INDEX "speech_embedding_speech_idx" ON "speech_embedding" USING btree ("speech_id");
|
||||
CREATE INDEX "speech_embedding_kind_idx" ON "speech_embedding" USING btree ("kind");
|
||||
CREATE INDEX "speech_embedding_updated_at_idx" ON "speech_embedding" USING btree ("updated_at");
|
||||
CREATE INDEX "speech_embedding_created_at_idx" ON "speech_embedding" USING btree ("created_at");
|
||||
ALTER TABLE "payload_locked_documents_rels" ADD CONSTRAINT "payload_locked_documents_rels_speech_embedding_fk" FOREIGN KEY ("speech_embedding_id") REFERENCES "public"."speech_embedding"("id") ON DELETE cascade ON UPDATE no action;
|
||||
CREATE INDEX "payload_locked_documents_rels_speech_embedding_id_idx" ON "payload_locked_documents_rels" USING btree ("speech_embedding_id");`)
|
||||
}
|
||||
|
||||
export async function down({ db }: MigrateDownArgs): Promise<void> {
|
||||
await db.execute(sql`
|
||||
ALTER TABLE "speech_embedding" DISABLE ROW LEVEL SECURITY;
|
||||
ALTER TABLE "payload_locked_documents_rels" DROP CONSTRAINT IF EXISTS "payload_locked_documents_rels_speech_embedding_fk";
|
||||
DROP INDEX IF EXISTS "payload_locked_documents_rels_speech_embedding_id_idx";
|
||||
ALTER TABLE "payload_locked_documents_rels" DROP COLUMN IF EXISTS "speech_embedding_id";
|
||||
DROP TABLE IF EXISTS "speech_embedding" CASCADE;
|
||||
DROP TYPE IF EXISTS "public"."enum_speech_embedding_kind";`)
|
||||
}
|
||||
|
|
@ -8023,6 +8023,163 @@
|
|||
"checkConstraints": {},
|
||||
"isRLSEnabled": false
|
||||
},
|
||||
"public.speech_embedding": {
|
||||
"name": "speech_embedding",
|
||||
"schema": "",
|
||||
"columns": {
|
||||
"id": {
|
||||
"name": "id",
|
||||
"type": "serial",
|
||||
"primaryKey": true,
|
||||
"notNull": true
|
||||
},
|
||||
"speech_id": {
|
||||
"name": "speech_id",
|
||||
"type": "integer",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"kind": {
|
||||
"name": "kind",
|
||||
"type": "enum_speech_embedding_kind",
|
||||
"typeSchema": "public",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"order": {
|
||||
"name": "order",
|
||||
"type": "numeric",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"start_seconds": {
|
||||
"name": "start_seconds",
|
||||
"type": "numeric",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"model": {
|
||||
"name": "model",
|
||||
"type": "varchar",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"dimensions": {
|
||||
"name": "dimensions",
|
||||
"type": "numeric",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"content_hash": {
|
||||
"name": "content_hash",
|
||||
"type": "varchar",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"vector": {
|
||||
"name": "vector",
|
||||
"type": "jsonb",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"updated_at": {
|
||||
"name": "updated_at",
|
||||
"type": "timestamp(3) with time zone",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": "now()"
|
||||
},
|
||||
"created_at": {
|
||||
"name": "created_at",
|
||||
"type": "timestamp(3) with time zone",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": "now()"
|
||||
}
|
||||
},
|
||||
"indexes": {
|
||||
"speech_embedding_speech_idx": {
|
||||
"name": "speech_embedding_speech_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "speech_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
},
|
||||
"speech_embedding_kind_idx": {
|
||||
"name": "speech_embedding_kind_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "kind",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
},
|
||||
"speech_embedding_updated_at_idx": {
|
||||
"name": "speech_embedding_updated_at_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "updated_at",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
},
|
||||
"speech_embedding_created_at_idx": {
|
||||
"name": "speech_embedding_created_at_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "created_at",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
}
|
||||
},
|
||||
"foreignKeys": {
|
||||
"speech_embedding_speech_id_speech_id_fk": {
|
||||
"name": "speech_embedding_speech_id_speech_id_fk",
|
||||
"tableFrom": "speech_embedding",
|
||||
"tableTo": "speech",
|
||||
"columnsFrom": [
|
||||
"speech_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"onDelete": "set null",
|
||||
"onUpdate": "no action"
|
||||
}
|
||||
},
|
||||
"compositePrimaryKeys": {},
|
||||
"uniqueConstraints": {},
|
||||
"policies": {},
|
||||
"checkConstraints": {},
|
||||
"isRLSEnabled": false
|
||||
},
|
||||
"public.internet_speech_media": {
|
||||
"name": "internet_speech_media",
|
||||
"schema": "",
|
||||
|
|
@ -16045,6 +16202,12 @@
|
|||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"speech_embedding_id": {
|
||||
"name": "speech_embedding_id",
|
||||
"type": "integer",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"internet_speech_media_id": {
|
||||
"name": "internet_speech_media_id",
|
||||
"type": "integer",
|
||||
|
|
@ -16557,6 +16720,21 @@
|
|||
"method": "btree",
|
||||
"with": {}
|
||||
},
|
||||
"payload_locked_documents_rels_speech_embedding_id_idx": {
|
||||
"name": "payload_locked_documents_rels_speech_embedding_id_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "speech_embedding_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
},
|
||||
"payload_locked_documents_rels_internet_speech_media_id_idx": {
|
||||
"name": "payload_locked_documents_rels_internet_speech_media_id_idx",
|
||||
"columns": [
|
||||
|
|
@ -17220,6 +17398,19 @@
|
|||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
},
|
||||
"payload_locked_documents_rels_speech_embedding_fk": {
|
||||
"name": "payload_locked_documents_rels_speech_embedding_fk",
|
||||
"tableFrom": "payload_locked_documents_rels",
|
||||
"tableTo": "speech_embedding",
|
||||
"columnsFrom": [
|
||||
"speech_embedding_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
},
|
||||
"payload_locked_documents_rels_internet_speech_media_fk": {
|
||||
"name": "payload_locked_documents_rels_internet_speech_media_fk",
|
||||
"tableFrom": "payload_locked_documents_rels",
|
||||
|
|
@ -19164,6 +19355,15 @@
|
|||
"publishing"
|
||||
]
|
||||
},
|
||||
"public.enum_speech_embedding_kind": {
|
||||
"name": "enum_speech_embedding_kind",
|
||||
"schema": "public",
|
||||
"values": [
|
||||
"speech",
|
||||
"segment",
|
||||
"window"
|
||||
]
|
||||
},
|
||||
"public.enum_reel_feature": {
|
||||
"name": "enum_reel_feature",
|
||||
"schema": "public",
|
||||
|
|
|
|||
|
|
@ -86,6 +86,7 @@ import * as migration_20260924_040210_add_recording_facets from './20260924_0402
|
|||
import * as migration_20260924_042028_add_web_speech_origin from './20260924_042028_add_web_speech_origin';
|
||||
import * as migration_20260924_045625_add_content_piece_link_failure_reason from './20260924_045625_add_content_piece_link_failure_reason';
|
||||
import * as migration_20260924_123940_add_content_piece_people from './20260924_123940_add_content_piece_people';
|
||||
import * as migration_20260926_034118_add_speech_embedding from './20260926_034118_add_speech_embedding';
|
||||
import * as migration_20260926_062039_add_archive_photo from './20260926_062039_add_archive_photo';
|
||||
|
||||
export const migrations = [
|
||||
|
|
@ -529,6 +530,11 @@ export const migrations = [
|
|||
down: migration_20260924_123940_add_content_piece_people.down,
|
||||
name: '20260924_123940_add_content_piece_people',
|
||||
},
|
||||
{
|
||||
up: migration_20260926_034118_add_speech_embedding.up,
|
||||
down: migration_20260926_034118_add_speech_embedding.down,
|
||||
name: '20260926_034118_add_speech_embedding'
|
||||
},
|
||||
{
|
||||
up: migration_20260926_062039_add_archive_photo.up,
|
||||
down: migration_20260926_062039_add_archive_photo.down,
|
||||
|
|
|
|||
|
|
@ -89,6 +89,7 @@ export interface Config {
|
|||
speech: Speech;
|
||||
speechSegment: SpeechSegment;
|
||||
speechCut: SpeechCut;
|
||||
speechEmbedding: SpeechEmbedding;
|
||||
internetSpeechMedia: InternetSpeechMedia;
|
||||
reel: Reel;
|
||||
reelMedia: ReelMedia;
|
||||
|
|
@ -145,6 +146,7 @@ export interface Config {
|
|||
speech: SpeechSelect<false> | SpeechSelect<true>;
|
||||
speechSegment: SpeechSegmentSelect<false> | SpeechSegmentSelect<true>;
|
||||
speechCut: SpeechCutSelect<false> | SpeechCutSelect<true>;
|
||||
speechEmbedding: SpeechEmbeddingSelect<false> | SpeechEmbeddingSelect<true>;
|
||||
internetSpeechMedia: InternetSpeechMediaSelect<false> | InternetSpeechMediaSelect<true>;
|
||||
reel: ReelSelect<false> | ReelSelect<true>;
|
||||
reelMedia: ReelMediaSelect<false> | ReelMediaSelect<true>;
|
||||
|
|
@ -1141,6 +1143,46 @@ export interface SpeechCut {
|
|||
updatedAt: string;
|
||||
createdAt: string;
|
||||
}
|
||||
/**
|
||||
* This interface was referenced by `Config`'s JSON-Schema
|
||||
* via the `definition` "speechEmbedding".
|
||||
*/
|
||||
export interface SpeechEmbedding {
|
||||
id: number;
|
||||
speech: number | Speech;
|
||||
kind: 'speech' | 'segment' | 'window';
|
||||
/**
|
||||
* Posição do trecho na fala; vazio no vetor da fala.
|
||||
*/
|
||||
order?: number | null;
|
||||
/**
|
||||
* Início do segmento quando conhecido; vazio na janela de texto.
|
||||
*/
|
||||
startSeconds?: number | null;
|
||||
/**
|
||||
* Modelo de embedding que gerou o vetor (muda ⇒ reindexar).
|
||||
*/
|
||||
model: string;
|
||||
dimensions: number;
|
||||
/**
|
||||
* Hash do texto + modelo; decide o pulo na reindexação.
|
||||
*/
|
||||
contentHash: string;
|
||||
/**
|
||||
* Embedding L2-normalizado da unidade.
|
||||
*/
|
||||
vector:
|
||||
| {
|
||||
[k: string]: unknown;
|
||||
}
|
||||
| unknown[]
|
||||
| string
|
||||
| number
|
||||
| boolean
|
||||
| null;
|
||||
updatedAt: string;
|
||||
createdAt: string;
|
||||
}
|
||||
/**
|
||||
* Reels de tutorial da assessoria. Os arquivos são privados e só abrem com login da campanha; publicar não dispara nada no Instagram.
|
||||
*
|
||||
|
|
@ -2230,6 +2272,10 @@ export interface PayloadLockedDocument {
|
|||
relationTo: 'speechCut';
|
||||
value: number | SpeechCut;
|
||||
} | null)
|
||||
| ({
|
||||
relationTo: 'speechEmbedding';
|
||||
value: number | SpeechEmbedding;
|
||||
} | null)
|
||||
| ({
|
||||
relationTo: 'internetSpeechMedia';
|
||||
value: number | InternetSpeechMedia;
|
||||
|
|
@ -2867,6 +2913,22 @@ export interface SpeechCutSelect<T extends boolean = true> {
|
|||
updatedAt?: T;
|
||||
createdAt?: T;
|
||||
}
|
||||
/**
|
||||
* This interface was referenced by `Config`'s JSON-Schema
|
||||
* via the `definition` "speechEmbedding_select".
|
||||
*/
|
||||
export interface SpeechEmbeddingSelect<T extends boolean = true> {
|
||||
speech?: T;
|
||||
kind?: T;
|
||||
order?: T;
|
||||
startSeconds?: T;
|
||||
model?: T;
|
||||
dimensions?: T;
|
||||
contentHash?: T;
|
||||
vector?: T;
|
||||
updatedAt?: T;
|
||||
createdAt?: T;
|
||||
}
|
||||
/**
|
||||
* This interface was referenced by `Config`'s JSON-Schema
|
||||
* via the `definition` "internetSpeechMedia_select".
|
||||
|
|
@ -3870,6 +3932,7 @@ export interface TaskCreateCollectionExport {
|
|||
| 'speech'
|
||||
| 'speechSegment'
|
||||
| 'speechCut'
|
||||
| 'speechEmbedding'
|
||||
| 'internetSpeechMedia'
|
||||
| 'reel'
|
||||
| 'reelMedia'
|
||||
|
|
|
|||
|
|
@ -47,6 +47,7 @@ import { ShareLink } from './collections/ShareLink'
|
|||
import { Signature } from './collections/Signature'
|
||||
import { Speech } from './collections/Speech'
|
||||
import { SpeechCut } from './collections/SpeechCut'
|
||||
import { SpeechEmbedding } from './collections/SpeechEmbedding'
|
||||
import { SpeechSegment } from './collections/SpeechSegment'
|
||||
import { StateDeputy } from './collections/StateDeputy'
|
||||
import { Subscription } from './collections/Subscription'
|
||||
|
|
@ -130,6 +131,7 @@ export default buildConfig({
|
|||
Speech,
|
||||
SpeechSegment,
|
||||
SpeechCut,
|
||||
SpeechEmbedding,
|
||||
InternetSpeechMedia,
|
||||
Reel,
|
||||
ReelMedia,
|
||||
|
|
|
|||
134
src/utilities/ai/deepInfraEmbed.ts
Normal file
134
src/utilities/ai/deepInfraEmbed.ts
Normal file
|
|
@ -0,0 +1,134 @@
|
|||
import 'server-only'
|
||||
|
||||
import { normalizeVector, type SpeechSemanticVector } from '@/lib/speechSemantic'
|
||||
|
||||
/**
|
||||
* C229 — the embeddings network seam of the speech acervo, deliberately the
|
||||
* only place the app/index CLI talks to Deep Infra for sense. Same provider and
|
||||
* key as the ASR (`deepInfraTranscribe.ts`), OpenAI-compatible endpoint and the
|
||||
* same contract: never throws — a missing key, timeout, provider error or
|
||||
* malformed output resolves to `null` and the caller degrades honestly.
|
||||
*
|
||||
* The model is BAAI/bge-m3 (multilingual, 1024 dims, 8192 tokens) at
|
||||
* US$0.010/1M tokens (DeepInfra public price, 2026-09-25); the vectors are
|
||||
* L2-normalized here so the index and the query share the same scale and the
|
||||
* pure ranking can stay a cosine.
|
||||
*/
|
||||
|
||||
export const DEEPINFRA_EMBED_URL = 'https://api.deepinfra.com/v1/openai/embeddings'
|
||||
export const DEEPINFRA_EMBED_MODEL = 'BAAI/bge-m3'
|
||||
export const DEEPINFRA_EMBED_DIMENSIONS = 1024
|
||||
export const DEEPINFRA_EMBED_COST_PER_MILLION_TOKENS_USD = 0.01
|
||||
|
||||
/** Cost guard per text: a window is ~1200 chars; anything longer is truncated. */
|
||||
export const EMBED_TEXT_MAX_CHARS = 2000
|
||||
export const EMBED_BATCH_SIZE = 32
|
||||
|
||||
const EMBED_TIMEOUT_MS = 30_000
|
||||
const QUERY_TIMEOUT_MS = 8_000
|
||||
|
||||
export type SpeechTextsEmbedding = {
|
||||
vectors: SpeechSemanticVector[]
|
||||
promptTokens: number
|
||||
}
|
||||
|
||||
/** The injectable query boundary (tests pass a deterministic resolver). */
|
||||
export type SpeechQueryEmbeddingResolver = (query: string) => Promise<SpeechSemanticVector | null>
|
||||
|
||||
type EmbedResponse = {
|
||||
data?: unknown
|
||||
usage?: { prompt_tokens?: unknown }
|
||||
}
|
||||
|
||||
/** Parses one OpenAI-shaped batch and returns the vectors in input order. */
|
||||
const parseBatch = (
|
||||
payload: unknown,
|
||||
expected: number,
|
||||
): { vectors: number[][]; promptTokens: number } | null => {
|
||||
const { data, usage } = (payload ?? {}) as EmbedResponse
|
||||
if (!Array.isArray(data) || data.length !== expected) return null
|
||||
|
||||
const byIndex = new Array<number[] | null>(expected).fill(null)
|
||||
for (const item of data) {
|
||||
if (!item || typeof item !== 'object') return null
|
||||
const { index, embedding } = item as { index?: unknown; embedding?: unknown }
|
||||
if (typeof index !== 'number' || !Number.isInteger(index) || index < 0 || index >= expected) {
|
||||
return null
|
||||
}
|
||||
if (!Array.isArray(embedding) || embedding.length !== DEEPINFRA_EMBED_DIMENSIONS) return null
|
||||
if (!embedding.every((value) => typeof value === 'number' && Number.isFinite(value))) {
|
||||
return null
|
||||
}
|
||||
byIndex[index] = embedding as number[]
|
||||
}
|
||||
if (byIndex.some((vector) => vector === null)) return null
|
||||
|
||||
const promptTokens = typeof usage?.prompt_tokens === 'number' ? usage.prompt_tokens : 0
|
||||
return { vectors: byIndex.map((vector) => normalizeVector(vector ?? [])), promptTokens }
|
||||
}
|
||||
|
||||
/**
|
||||
* Embeds texts in `EMBED_BATCH_SIZE` batches, in input order. Returns `null`
|
||||
* when the mechanism is unavailable or any batch comes back malformed — the
|
||||
* index CLI never persists a partial/mixed index from a failed run.
|
||||
*/
|
||||
export const embedSpeechTexts = async (
|
||||
texts: readonly string[],
|
||||
{ timeoutMs = EMBED_TIMEOUT_MS }: { timeoutMs?: number } = {},
|
||||
): Promise<SpeechTextsEmbedding | null> => {
|
||||
if (texts.length === 0) return { vectors: [], promptTokens: 0 }
|
||||
|
||||
const apiKey = process.env.DEEPINFRA_API_KEY
|
||||
if (!apiKey) return null
|
||||
|
||||
const batches: string[][] = []
|
||||
for (let start = 0; start < texts.length; start += EMBED_BATCH_SIZE) {
|
||||
batches.push(
|
||||
texts
|
||||
.slice(start, start + EMBED_BATCH_SIZE)
|
||||
.map((text) => text.slice(0, EMBED_TEXT_MAX_CHARS)),
|
||||
)
|
||||
}
|
||||
|
||||
const vectors: number[][] = []
|
||||
let promptTokens = 0
|
||||
for (const input of batches) {
|
||||
try {
|
||||
const response = await fetch(DEEPINFRA_EMBED_URL, {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'Content-Type': 'application/json',
|
||||
Authorization: `Bearer ${apiKey}`,
|
||||
},
|
||||
body: JSON.stringify({
|
||||
model: DEEPINFRA_EMBED_MODEL,
|
||||
input,
|
||||
encoding_format: 'float',
|
||||
}),
|
||||
signal: AbortSignal.timeout(timeoutMs),
|
||||
})
|
||||
if (!response.ok) return null
|
||||
|
||||
const parsed = parseBatch(await response.json(), input.length)
|
||||
if (!parsed) return null
|
||||
vectors.push(...parsed.vectors)
|
||||
promptTokens += parsed.promptTokens
|
||||
} catch {
|
||||
return null
|
||||
}
|
||||
}
|
||||
|
||||
return { vectors, promptTokens }
|
||||
}
|
||||
|
||||
/**
|
||||
* The single query embedding of a theme search. A blank query or an
|
||||
* unavailable provider resolves to `null` (the loader degrades to the literal
|
||||
* search with the discreet notice).
|
||||
*/
|
||||
export const embedSpeechQuery: SpeechQueryEmbeddingResolver = async (query) => {
|
||||
const trimmed = query.trim()
|
||||
if (!trimmed) return null
|
||||
const embedded = await embedSpeechTexts([trimmed], { timeoutMs: QUERY_TIMEOUT_MS })
|
||||
return embedded?.vectors[0] ?? null
|
||||
}
|
||||
|
|
@ -269,7 +269,7 @@ const toExcerptViewModel = ({ candidate, motivo }: SelectedCandidate) => ({
|
|||
...(motivo ? { motivo } : {}),
|
||||
url: buildWatchHref(
|
||||
candidate.speechId,
|
||||
candidate.excerpt,
|
||||
candidate.excerpt.startSeconds,
|
||||
candidate.excerpt.matchedTerms.join(' '),
|
||||
),
|
||||
})
|
||||
|
|
|
|||
313
src/utilities/speech/speechEmbeddingIndex.ts
Normal file
313
src/utilities/speech/speechEmbeddingIndex.ts
Normal file
|
|
@ -0,0 +1,313 @@
|
|||
import 'server-only'
|
||||
|
||||
import type { Payload } from 'payload'
|
||||
|
||||
import {
|
||||
cosineSimilarity,
|
||||
meanPoolVectors,
|
||||
planSpeechIndex,
|
||||
type SpeechIndexPlan,
|
||||
type SpeechIndexSegment,
|
||||
} from '@/lib/speechSemantic'
|
||||
import { DEEPINFRA_EMBED_MODEL, embedSpeechTexts } from '@/utilities/ai/deepInfraEmbed'
|
||||
import { withPayloadTransaction } from '@/utilities/payloadTransaction'
|
||||
|
||||
/**
|
||||
* C229 — the index owner: plans the retrieval units of a speech (pure), embeds
|
||||
* the stale ones in one provider call and replaces the rows in a transaction.
|
||||
* The CLI is the only production caller (a trusted actor, `overrideAccess:
|
||||
* true`); the loader only reads. Idempotent by the speech-level `contentHash`
|
||||
* (unit texts + model), so a re-run after an import only pays for what changed.
|
||||
*/
|
||||
|
||||
export type SpeechIndexSource = {
|
||||
id: number
|
||||
officialTranscript?: string | null
|
||||
summary?: string | null
|
||||
searchText?: string | null
|
||||
segments: readonly SpeechIndexSegment[]
|
||||
}
|
||||
|
||||
type PlannedSpeechIndex = {
|
||||
source: SpeechIndexSource
|
||||
plan: SpeechIndexPlan
|
||||
}
|
||||
|
||||
export type SpeechIndexPlanBatch = {
|
||||
/** Speeches whose units must be embedded (hash/model changed). */
|
||||
stale: PlannedSpeechIndex[]
|
||||
/** Speech-level hash already matches the plan. */
|
||||
upToDate: number
|
||||
/** No usable text at all. */
|
||||
skippedNoText: number
|
||||
units: number
|
||||
}
|
||||
|
||||
export type SpeechIndexBatchResult = {
|
||||
indexed: number
|
||||
upToDate: number
|
||||
skippedNoText: number
|
||||
failed: number
|
||||
units: number
|
||||
promptTokens: number
|
||||
}
|
||||
|
||||
export type SpeechIndexPlanDeps = {
|
||||
force?: boolean
|
||||
}
|
||||
|
||||
export type SpeechIndexDeps = SpeechIndexPlanDeps & {
|
||||
embedTexts?: typeof embedSpeechTexts
|
||||
}
|
||||
|
||||
/** Stored speech-level hashes of the given speeches (the skip key). */
|
||||
const loadStoredSpeechHashes = async (
|
||||
payload: Payload,
|
||||
speechIds: readonly number[],
|
||||
): Promise<Map<number, string>> => {
|
||||
const stored = new Map<number, string>()
|
||||
if (speechIds.length === 0) return stored
|
||||
|
||||
const result = await payload.find({
|
||||
collection: 'speechEmbedding',
|
||||
where: { and: [{ kind: { equals: 'speech' } }, { speech: { in: [...speechIds] } }] },
|
||||
depth: 0,
|
||||
limit: 0,
|
||||
pagination: false,
|
||||
select: { speech: true, contentHash: true },
|
||||
// Intentional bypass: the index CLI is a trusted actor with no session.
|
||||
overrideAccess: true,
|
||||
})
|
||||
|
||||
for (const row of result.docs) {
|
||||
const speech = row.speech
|
||||
const speechId = typeof speech === 'number' ? speech : (speech?.id ?? null)
|
||||
if (typeof speechId === 'number' && row.contentHash) stored.set(speechId, row.contentHash)
|
||||
}
|
||||
return stored
|
||||
}
|
||||
|
||||
/** Plans a batch and separates what must be embedded from what is current. */
|
||||
export const planSpeechIndexBatch = async (
|
||||
payload: Payload,
|
||||
sources: readonly SpeechIndexSource[],
|
||||
{ force = false }: SpeechIndexPlanDeps = {},
|
||||
): Promise<SpeechIndexPlanBatch> => {
|
||||
const planned: PlannedSpeechIndex[] = []
|
||||
let skippedNoText = 0
|
||||
|
||||
for (const source of sources) {
|
||||
const plan = planSpeechIndex(source, { model: DEEPINFRA_EMBED_MODEL })
|
||||
if (!plan) {
|
||||
skippedNoText += 1
|
||||
continue
|
||||
}
|
||||
planned.push({ source, plan })
|
||||
}
|
||||
|
||||
const storedHashes = force
|
||||
? new Map<number, string>()
|
||||
: await loadStoredSpeechHashes(
|
||||
payload,
|
||||
planned.map(({ source }) => source.id),
|
||||
)
|
||||
|
||||
const stale = planned.filter(
|
||||
({ source, plan }) => storedHashes.get(source.id) !== plan.speechHash,
|
||||
)
|
||||
return {
|
||||
stale,
|
||||
upToDate: planned.length - stale.length,
|
||||
skippedNoText,
|
||||
units: stale.reduce((total, { plan }) => total + plan.units.length, 0),
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Embeds and persists the stale speeches of a batch. The provider call is ONE
|
||||
* ordered sequence for the whole batch: a provider failure writes nothing.
|
||||
* Persistence is per speech (one transaction each), so a database failure
|
||||
* mid-batch can leave the earlier speeches indexed — `failed` reports it
|
||||
* honestly and the next run only pays for the rest.
|
||||
*/
|
||||
export const indexSpeechSources = async (
|
||||
payload: Payload,
|
||||
sources: readonly SpeechIndexSource[],
|
||||
deps: SpeechIndexDeps = {},
|
||||
): Promise<SpeechIndexBatchResult> => {
|
||||
const { embedTexts = embedSpeechTexts, force = false } = deps
|
||||
const batch = await planSpeechIndexBatch(payload, sources, { force })
|
||||
const base = {
|
||||
upToDate: batch.upToDate,
|
||||
skippedNoText: batch.skippedNoText,
|
||||
units: 0,
|
||||
promptTokens: 0,
|
||||
}
|
||||
if (batch.stale.length === 0) return { ...base, indexed: 0, failed: 0 }
|
||||
|
||||
const texts = batch.stale.flatMap(({ plan }) => plan.units.map((unit) => unit.text))
|
||||
const embedded = await embedTexts(texts)
|
||||
if (!embedded) {
|
||||
return { ...base, indexed: 0, failed: batch.stale.length }
|
||||
}
|
||||
// The whole batch was embedded already: the tokens were spent even if a
|
||||
// later write fails.
|
||||
const spent = { units: texts.length, promptTokens: embedded.promptTokens }
|
||||
|
||||
let cursor = 0
|
||||
let indexed = 0
|
||||
for (const { source, plan } of batch.stale) {
|
||||
const vectors = embedded.vectors.slice(cursor, cursor + plan.units.length)
|
||||
cursor += plan.units.length
|
||||
if (vectors.length !== plan.units.length) {
|
||||
return { ...base, ...spent, indexed, failed: batch.stale.length - indexed }
|
||||
}
|
||||
|
||||
const speechVector = meanPoolVectors(vectors)
|
||||
await withPayloadTransaction(payload, async ({ req }) => {
|
||||
await payload.delete({
|
||||
collection: 'speechEmbedding',
|
||||
where: { speech: { equals: source.id } },
|
||||
req,
|
||||
// Intentional bypass: rebuilding the derived index of a trusted run.
|
||||
overrideAccess: true,
|
||||
})
|
||||
await payload.create({
|
||||
collection: 'speechEmbedding',
|
||||
data: {
|
||||
speech: source.id,
|
||||
kind: 'speech',
|
||||
model: DEEPINFRA_EMBED_MODEL,
|
||||
dimensions: speechVector.length,
|
||||
contentHash: plan.speechHash,
|
||||
vector: Array.from(speechVector),
|
||||
},
|
||||
req,
|
||||
// Intentional bypass: same trusted rebuild.
|
||||
overrideAccess: true,
|
||||
})
|
||||
for (const [index, unit] of plan.units.entries()) {
|
||||
const vector = vectors[index]
|
||||
if (!vector) continue
|
||||
await payload.create({
|
||||
collection: 'speechEmbedding',
|
||||
data: {
|
||||
speech: source.id,
|
||||
kind: unit.kind,
|
||||
order: unit.order,
|
||||
startSeconds: unit.startSeconds,
|
||||
model: DEEPINFRA_EMBED_MODEL,
|
||||
dimensions: vector.length,
|
||||
contentHash: `${plan.speechHash}#${unit.kind}:${unit.order}`,
|
||||
vector: Array.from(vector),
|
||||
},
|
||||
req,
|
||||
// Intentional bypass: same trusted rebuild.
|
||||
overrideAccess: true,
|
||||
})
|
||||
}
|
||||
})
|
||||
indexed += 1
|
||||
}
|
||||
|
||||
return {
|
||||
upToDate: base.upToDate,
|
||||
skippedNoText: base.skippedNoText,
|
||||
indexed,
|
||||
failed: batch.stale.length - indexed,
|
||||
...spent,
|
||||
}
|
||||
}
|
||||
|
||||
export type SpeechIndexProbeHit = {
|
||||
id: number
|
||||
sourceKey: string
|
||||
speechAt: string
|
||||
origin: string
|
||||
score: number
|
||||
}
|
||||
|
||||
/**
|
||||
* Read-only calibration of the theme cut-off: ranks every indexed speech
|
||||
* against the query vector (score included) and returns the top N. The score
|
||||
* never reaches the UI; this is the CLI-only instrument the runbook uses to
|
||||
* verify the acceptance themes and tune `SPEECH_SEMANTIC_MIN_COSINE`.
|
||||
*/
|
||||
export const probeSpeechIndex = async (
|
||||
payload: Payload,
|
||||
queryVector: readonly number[],
|
||||
{ limit = 10 }: { limit?: number } = {},
|
||||
): Promise<SpeechIndexProbeHit[]> => {
|
||||
const embeddings = await payload.find({
|
||||
collection: 'speechEmbedding',
|
||||
// Same embedding space as the query: another model's 1024 dims would score
|
||||
// nonsense (the collection keeps `model` exactly to denounce that).
|
||||
where: { and: [{ kind: { equals: 'speech' } }, { model: { equals: DEEPINFRA_EMBED_MODEL } }] },
|
||||
depth: 0,
|
||||
limit: 0,
|
||||
pagination: false,
|
||||
select: { speech: true, vector: true },
|
||||
// Intentional bypass: the probe is a trusted read with no session.
|
||||
overrideAccess: true,
|
||||
})
|
||||
|
||||
const vectorBySpeech = new Map<number, number[]>()
|
||||
for (const row of embeddings.docs) {
|
||||
const speech = row.speech
|
||||
const speechId = typeof speech === 'number' ? speech : (speech?.id ?? null)
|
||||
const vector = Array.isArray(row.vector) ? row.vector : null
|
||||
if (typeof speechId !== 'number' || !vector || vector.length !== queryVector.length) continue
|
||||
if (!vector.every((value) => typeof value === 'number' && Number.isFinite(value))) continue
|
||||
vectorBySpeech.set(speechId, vector as number[])
|
||||
}
|
||||
if (vectorBySpeech.size === 0) return []
|
||||
|
||||
const speeches = await payload.find({
|
||||
collection: 'speech',
|
||||
where: { id: { in: [...vectorBySpeech.keys()] } },
|
||||
depth: 0,
|
||||
limit: 0,
|
||||
pagination: false,
|
||||
select: { sourceKey: true, speechAt: true, origin: true },
|
||||
// Intentional bypass: same trusted read.
|
||||
overrideAccess: true,
|
||||
})
|
||||
|
||||
return speeches.docs
|
||||
.flatMap((speech) => {
|
||||
const vector = vectorBySpeech.get(speech.id)
|
||||
if (!vector) return []
|
||||
const score = cosineSimilarity(queryVector, vector)
|
||||
return [
|
||||
{
|
||||
id: speech.id,
|
||||
sourceKey: speech.sourceKey,
|
||||
speechAt: speech.speechAt,
|
||||
origin: speech.origin,
|
||||
score,
|
||||
},
|
||||
]
|
||||
})
|
||||
.sort((left, right) => right.score - left.score)
|
||||
.slice(0, Math.max(1, limit))
|
||||
}
|
||||
|
||||
/** Shape shared by the CLI report and the `--dry-run` preview. */
|
||||
export type SpeechIndexPreview = {
|
||||
speeches: number
|
||||
upToDate: number
|
||||
skippedNoText: number
|
||||
toIndex: number
|
||||
units: number
|
||||
}
|
||||
|
||||
export const previewSpeechIndex = (
|
||||
batch: SpeechIndexPlanBatch,
|
||||
speeches: number,
|
||||
): SpeechIndexPreview => ({
|
||||
speeches,
|
||||
upToDate: batch.upToDate,
|
||||
skippedNoText: batch.skippedNoText,
|
||||
toIndex: batch.stale.length,
|
||||
units: batch.units,
|
||||
})
|
||||
|
|
@ -36,7 +36,13 @@ export const durationBucketWhere = (bucket: SpeechDurationBucket): Where => {
|
|||
}
|
||||
}
|
||||
|
||||
const buildSpeechFacetWhere = (state: SpeechListState): Where[] => {
|
||||
/**
|
||||
* C229 — the facet-only branches of the acervo. This is the ONE candidate
|
||||
* boundary of the semantic theme path: the sense search ranks every speech the
|
||||
* facets allow, attaching no textual branch (the `q` LIKE stays exclusive to
|
||||
* the literal/degraded path). Also the base of the literal `where` below.
|
||||
*/
|
||||
export const buildSpeechFacetWhere = (state: SpeechListState): Where[] => {
|
||||
// C215/C216 — ONE discriminator decides the source: the web list carries
|
||||
// `source=internet` and reads `origin: web`; everything else is the Câmara
|
||||
// list (the web speeches are never mixed here). The Fase facet is Câmara-only.
|
||||
|
|
@ -62,48 +68,28 @@ const buildSpeechFacetWhere = (state: SpeechListState): Where[] => {
|
|||
return filters
|
||||
}
|
||||
|
||||
const buildSpeechTextBranches = (q: string, themeTerms: readonly string[] = []): Where[] => {
|
||||
const buildSpeechTextBranches = (q: string): Where[] => {
|
||||
const branches: Where[] = []
|
||||
const seenText = new Set<string>()
|
||||
const seenKeyword = new Set<string>()
|
||||
|
||||
// C192 — the literal query plus the expanded theme terms, each contributing
|
||||
// the same two branches (normalized text / raw keyword). Duplicates are
|
||||
// dropped so an expansion that repeats the query does not widen the OR.
|
||||
for (const term of [q, ...themeTerms]) {
|
||||
const normalized = normalizeForSearch(term)
|
||||
if (normalized && !seenText.has(normalized)) {
|
||||
seenText.add(normalized)
|
||||
branches.push({ searchText: { like: normalized } })
|
||||
}
|
||||
const raw = term.trim()
|
||||
const lowered = raw.toLowerCase()
|
||||
if (raw && !seenKeyword.has(lowered)) {
|
||||
seenKeyword.add(lowered)
|
||||
branches.push({ keywords: { contains: raw } })
|
||||
}
|
||||
}
|
||||
|
||||
const normalized = normalizeForSearch(q)
|
||||
if (normalized) branches.push({ searchText: { like: normalized } })
|
||||
const raw = q.trim()
|
||||
if (raw) branches.push({ keywords: { contains: raw } })
|
||||
return branches
|
||||
}
|
||||
|
||||
/**
|
||||
* The textual branch: normalized speech text OR a raw official keyword, for the
|
||||
* query and (C192) every expanded theme term. The keyword `contains` compiles
|
||||
* to ILIKE `%q%` — case-insensitive, accent still significant. The view-model
|
||||
* mirror is `lib/speechSearch.speechMatchesSearchTerm`; a semantics change here
|
||||
* has to land there too.
|
||||
* The textual branch: normalized speech text OR a raw official keyword. The
|
||||
* keyword `contains` compiles to ILIKE `%q%` — case-insensitive, accent still
|
||||
* significant. The view-model mirror is `lib/speechSearch.speechMatchesSearchTerm`;
|
||||
* a semantics change here has to land there too.
|
||||
*/
|
||||
const buildSpeechTextWhere = (q: string, themeTerms: readonly string[] = []): Where => ({
|
||||
or: buildSpeechTextBranches(q, themeTerms),
|
||||
const buildSpeechTextWhere = (q: string): Where => ({
|
||||
or: buildSpeechTextBranches(q),
|
||||
})
|
||||
|
||||
export const buildSpeechListWhere = (
|
||||
state: SpeechListState,
|
||||
themeTerms: readonly string[] = [],
|
||||
): Where => {
|
||||
export const buildSpeechListWhere = (state: SpeechListState): Where => {
|
||||
const filters = buildSpeechFacetWhere(state)
|
||||
if (state.q) filters.push(buildSpeechTextWhere(state.q, themeTerms))
|
||||
if (state.q) filters.push(buildSpeechTextWhere(state.q))
|
||||
|
||||
return filters.length ? { and: filters } : {}
|
||||
}
|
||||
|
|
@ -111,18 +97,17 @@ export const buildSpeechListWhere = (
|
|||
/**
|
||||
* C174 (option B) — the acervo `where` widened to the origin speech of a
|
||||
* matching cut: the page still paginates by SPEECH, so the origin ids are OR-ed
|
||||
* into the textual branch (facets stay AND-ed over every row). C192 adds the
|
||||
* expanded theme terms to that same textual branch; only the literal query
|
||||
* drives the cut-origin lookup.
|
||||
* into the textual branch (facets stay AND-ed over every row). The cut-origin
|
||||
* lookup is lexical (`state.q`) and therefore never runs in the C229 theme
|
||||
* path — there the semantic engine selects the candidates.
|
||||
*/
|
||||
export const buildSpeechListWhereIncludingCutOrigins = (
|
||||
state: SpeechListState,
|
||||
originSpeechIds: readonly number[],
|
||||
themeTerms: readonly string[] = [],
|
||||
): Where => {
|
||||
const filters = buildSpeechFacetWhere(state)
|
||||
if (state.q) {
|
||||
const branches = buildSpeechTextBranches(state.q, themeTerms)
|
||||
const branches = buildSpeechTextBranches(state.q)
|
||||
if (originSpeechIds.length) branches.push({ id: { in: [...originSpeechIds] } })
|
||||
filters.push({ or: branches })
|
||||
}
|
||||
|
|
|
|||
|
|
@ -5,7 +5,12 @@
|
|||
* member equals no filter); year/phase/municipality are data-driven and
|
||||
* validated structurally.
|
||||
*/
|
||||
import { parseAcervoSort, type AcervoSortKey } from '@/lib/acervoListSort'
|
||||
import {
|
||||
acervoSortIsDuration,
|
||||
parseAcervoSort,
|
||||
parseAcervoThemeSort,
|
||||
type AcervoSortKey,
|
||||
} from '@/lib/acervoListSort'
|
||||
import { ACERVO_SOURCE_INTERNET, parseAcervoSource } from '@/lib/acervoSource'
|
||||
import { CAMPAIGN_COMMUNICATION_ACERVO } from '@/lib/campaignPaths'
|
||||
import { isContactSearchQueryReady } from '@/lib/contactSearchQuery'
|
||||
|
|
@ -31,10 +36,10 @@ import {
|
|||
export const speechPageSize = 25
|
||||
|
||||
/**
|
||||
* C192 — search mode of the acervo. `termo` is the literal search (default,
|
||||
* never serialized so existing deep links stay byte-identical); `tema` asks the
|
||||
* semantic bridge for related terms. Kept as a single-value param, only
|
||||
* meaningful alongside `q`.
|
||||
* C192/C229 — search mode of the acervo. `termo` is the literal search
|
||||
* (default, never serialized so existing deep links stay byte-identical);
|
||||
* `tema` asks the sense engine (C229) for related speeches. Kept as a
|
||||
* single-value param, only meaningful alongside `q`.
|
||||
*/
|
||||
export type SpeechSearchMode = 'termo' | 'tema'
|
||||
|
||||
|
|
@ -76,7 +81,11 @@ export type SpeechListState = {
|
|||
phases?: string[]
|
||||
municipalities?: number[]
|
||||
durations?: SpeechDurationBucket[]
|
||||
/** C216 — the web list ordering; the Câmara contract has no sort. */
|
||||
/**
|
||||
* C216 — the web list ordering; the Câmara contract has no sort. C229 — in
|
||||
* the theme mode the relevance order is the default (`sort` absent) and
|
||||
* `recentes` becomes a real state; the exact mode keeps its bytes.
|
||||
*/
|
||||
sort?: AcervoSortKey
|
||||
}
|
||||
|
||||
|
|
@ -135,7 +144,14 @@ export const parseSpeechListParams = (params: RawSpeechListParams): SpeechListSt
|
|||
params.duration,
|
||||
speechDurationSet,
|
||||
)
|
||||
const sort = isWeb ? parseAcervoSort(params.sort) : undefined
|
||||
// C229 — the web sort is mode-aware: the theme mode defaults to relevance
|
||||
// (`sort` absent) and recognizes `recentes`/`relevancia`; the exact mode
|
||||
// keeps the C216 rule (only the duration orders are meaningful).
|
||||
const sort = isWeb
|
||||
? mode === 'tema'
|
||||
? parseAcervoThemeSort(params.sort)
|
||||
: parseAcervoSort(params.sort)
|
||||
: undefined
|
||||
|
||||
return {
|
||||
...(isWeb ? { source: ACERVO_SOURCE_INTERNET } : {}),
|
||||
|
|
@ -189,9 +205,14 @@ export const serializeCanonicalSpeechListSearchParams = (
|
|||
params.append('municipality', String(municipality))
|
||||
}
|
||||
for (const duration of canonicalState.durations ?? []) params.append('duration', duration)
|
||||
// C216 — `recentes` is the default and never serialized; the sort belongs to
|
||||
// the web source only.
|
||||
if (canonicalState.source === 'internet' && canonicalState.sort) {
|
||||
// C216 — `recentes` is the exact-mode default and never serialized; C229 —
|
||||
// in the theme mode relevance took that place, so an explicit `recentes` is
|
||||
// a real state and serializes. Both rules live on the web source only.
|
||||
if (
|
||||
canonicalState.source === 'internet' &&
|
||||
canonicalState.sort &&
|
||||
(canonicalState.mode === 'tema' || acervoSortIsDuration(canonicalState.sort))
|
||||
) {
|
||||
params.set('sort', canonicalState.sort)
|
||||
}
|
||||
if (canonicalState.page > 1) params.set('page', String(canonicalState.page))
|
||||
|
|
|
|||
|
|
@ -3,12 +3,10 @@ import 'server-only'
|
|||
import type { Payload } from 'payload'
|
||||
|
||||
import { canReadCommunicationCatalog } from '@/lib/campaignRoles'
|
||||
import { speechSemanticSourceText, type SpeechSemanticVector } from '@/lib/speechSemantic'
|
||||
import { measuredVideoLagSeconds, parseYoutubeVideoId } from '@/lib/speechVod'
|
||||
import type { CampaignUser, Speech } from '@/payload-types'
|
||||
import {
|
||||
expandSpeechSearchTheme,
|
||||
type SpeechThemeExpansionResolver,
|
||||
} from '@/utilities/ai/expandSpeechSearchTheme'
|
||||
import { embedSpeechQuery, type SpeechQueryEmbeddingResolver } from '@/utilities/ai/deepInfraEmbed'
|
||||
import { createEntityNotFoundError } from '@/utilities/entityNotFound'
|
||||
import { loadMunicipalityLabelsByIds } from '@/utilities/loadNamesByIds'
|
||||
import { getYouTubeVideoStart } from '@/utilities/socialFeed/youtubeFeed'
|
||||
|
|
@ -27,6 +25,10 @@ import {
|
|||
type SpeechFilterOptions,
|
||||
type SpeechListState,
|
||||
} from '@/utilities/speech/speechListUrl'
|
||||
import {
|
||||
loadSpeechSemanticEvidence,
|
||||
searchSpeechesByMeaning,
|
||||
} from '@/utilities/speech/speechSemanticSearch'
|
||||
import {
|
||||
municipalityIdsOfSpeech,
|
||||
toSpeechDetailViewModel,
|
||||
|
|
@ -36,7 +38,8 @@ import {
|
|||
type SpeechDetailRecord,
|
||||
type SpeechDetailViewModel,
|
||||
type SpeechListItemViewModel,
|
||||
type SpeechSegmentRecord,
|
||||
type SpeechSegmentRecordWithOrder,
|
||||
type SpeechSemanticEvidence,
|
||||
type WebSpeechDetailRecord,
|
||||
type WebSpeechDetailViewModel,
|
||||
type WebSpeechListItemViewModel,
|
||||
|
|
@ -77,6 +80,7 @@ const speechDetailSelect = {
|
|||
|
||||
const segmentSelect = {
|
||||
speech: true,
|
||||
order: true,
|
||||
startSeconds: true,
|
||||
endSeconds: true,
|
||||
text: true,
|
||||
|
|
@ -134,36 +138,12 @@ const loadSpeechFilterOptions = async (
|
|||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* C216 — the shared theme gate of both sources: expands only for an actor who
|
||||
* may read the catalog (the `find` is still the final barrier) and only with a
|
||||
* query. The caller injects the resolver so the tests stay offline.
|
||||
*/
|
||||
const resolveSpeechThemeTerms = async ({
|
||||
state,
|
||||
role,
|
||||
expandTheme,
|
||||
}: {
|
||||
state: SpeechListState
|
||||
role: CampaignUser['role']
|
||||
expandTheme: SpeechThemeExpansionResolver
|
||||
}): Promise<{ themeTerms: readonly string[]; themeUnavailable: boolean }> => {
|
||||
const themeRequested = state.mode === 'tema' && Boolean(state.q)
|
||||
if (!themeRequested || !canReadCommunicationCatalog(role)) {
|
||||
return { themeTerms: [], themeUnavailable: false }
|
||||
}
|
||||
const expansion = await expandTheme(state.q ?? '')
|
||||
return expansion
|
||||
? { themeTerms: expansion.terms, themeUnavailable: false }
|
||||
: { themeTerms: [], themeUnavailable: true }
|
||||
}
|
||||
|
||||
export const loadSegmentsForSpeeches = async (
|
||||
payload: Payload,
|
||||
user: CampaignUser,
|
||||
speechIds: readonly number[],
|
||||
): Promise<Map<number, SpeechSegmentRecord[]>> => {
|
||||
const bySpeech = new Map<number, SpeechSegmentRecord[]>()
|
||||
): Promise<Map<number, SpeechSegmentRecordWithOrder[]>> => {
|
||||
const bySpeech = new Map<number, SpeechSegmentRecordWithOrder[]>()
|
||||
if (speechIds.length === 0) return bySpeech
|
||||
|
||||
const result = await payload.find({
|
||||
|
|
@ -182,6 +162,7 @@ export const loadSegmentsForSpeeches = async (
|
|||
const speechId = typeof segment.speech === 'number' ? segment.speech : segment.speech.id
|
||||
const list = bySpeech.get(speechId) ?? []
|
||||
list.push({
|
||||
order: segment.order,
|
||||
startSeconds: segment.startSeconds,
|
||||
endSeconds: segment.endSeconds,
|
||||
text: segment.text,
|
||||
|
|
@ -191,6 +172,108 @@ export const loadSegmentsForSpeeches = async (
|
|||
return bySpeech
|
||||
}
|
||||
|
||||
/**
|
||||
* C229 — the full docs of a page of semantic hits, in the ranking order (a
|
||||
* Payload `in` does not preserve it) and through the access layer.
|
||||
*/
|
||||
const loadSpeechDocsByIds = async (
|
||||
payload: Payload,
|
||||
user: CampaignUser,
|
||||
ids: readonly number[],
|
||||
select: Record<string, boolean>,
|
||||
): Promise<Speech[]> => {
|
||||
if (ids.length === 0) return []
|
||||
const result = await payload.find({
|
||||
collection: 'speech',
|
||||
where: { id: { in: [...ids] } },
|
||||
depth: 0,
|
||||
limit: 0,
|
||||
pagination: false,
|
||||
select,
|
||||
user,
|
||||
overrideAccess: false,
|
||||
})
|
||||
const byId = new Map(result.docs.map((doc) => [doc.id, doc as Speech]))
|
||||
return ids.flatMap((id) => {
|
||||
const doc = byId.get(id)
|
||||
return doc ? [doc] : []
|
||||
})
|
||||
}
|
||||
|
||||
type SemanticSpeechPage = {
|
||||
speeches: Speech[]
|
||||
segmentsBySpeech: Map<number, SpeechSegmentRecordWithOrder[]>
|
||||
evidenceBySpeech: Map<number, SpeechSemanticEvidence>
|
||||
totalDocs: number
|
||||
totalPages: number
|
||||
state: SpeechListState
|
||||
redirectHref?: string
|
||||
}
|
||||
|
||||
/**
|
||||
* C229 — the sense page shared by both sources: rank the facet-allowed
|
||||
* candidates, slice the requested page in memory, load the full docs in
|
||||
* ranking order and resolve the real passage closest to the theme. Returns
|
||||
* `null` when the engine must degrade (there are candidates but none is
|
||||
* indexed); a candidate-less combination is an honest empty page, not a
|
||||
* degradation — the engine ran and there is nothing to match.
|
||||
*/
|
||||
const loadSemanticSpeechPage = async ({
|
||||
payload,
|
||||
user,
|
||||
state,
|
||||
rawSearchParams,
|
||||
queryVector,
|
||||
select,
|
||||
}: {
|
||||
payload: Payload
|
||||
user: CampaignUser
|
||||
state: SpeechListState
|
||||
rawSearchParams: SpeechListSearchParams
|
||||
queryVector: SpeechSemanticVector
|
||||
select: Record<string, boolean>
|
||||
}): Promise<SemanticSpeechPage | null> => {
|
||||
const { hits, candidates, indexedCandidates } = await searchSpeechesByMeaning(
|
||||
payload,
|
||||
user,
|
||||
state,
|
||||
queryVector,
|
||||
)
|
||||
if (candidates > 0 && indexedCandidates === 0) return null
|
||||
|
||||
const totalPages = Math.max(1, Math.ceil(hits.length / speechPageSize))
|
||||
const resolvedUrl = resolveSpeechListUrl(rawSearchParams, totalPages)
|
||||
const page = resolvedUrl.state.page
|
||||
const pageHits = hits.slice((page - 1) * speechPageSize, page * speechPageSize)
|
||||
const speeches = await loadSpeechDocsByIds(
|
||||
payload,
|
||||
user,
|
||||
pageHits.map((hit) => hit.id),
|
||||
select,
|
||||
)
|
||||
const speechIds = speeches.map((speech) => speech.id)
|
||||
|
||||
const segmentsBySpeech = await loadSegmentsForSpeeches(payload, user, speechIds)
|
||||
const evidenceBySpeech = await loadSpeechSemanticEvidence(
|
||||
payload,
|
||||
user,
|
||||
queryVector,
|
||||
speechIds,
|
||||
segmentsBySpeech,
|
||||
new Map(speeches.map((speech) => [speech.id, speechSemanticSourceText(speech)])),
|
||||
)
|
||||
|
||||
return {
|
||||
speeches,
|
||||
segmentsBySpeech,
|
||||
evidenceBySpeech,
|
||||
totalDocs: hits.length,
|
||||
totalPages,
|
||||
state: resolvedUrl.state,
|
||||
redirectHref: resolvedUrl.redirectHref,
|
||||
}
|
||||
}
|
||||
|
||||
export type SpeechAcervoPageData = {
|
||||
rows: SpeechListItemViewModel[]
|
||||
state: SpeechListState
|
||||
|
|
@ -199,11 +282,12 @@ export type SpeechAcervoPageData = {
|
|||
totalPages: number
|
||||
filterOptions: SpeechFilterOptions
|
||||
/**
|
||||
* C192 — true when the actor asked for `mode=tema` but the expansion
|
||||
* mechanism was unavailable; the page shows the discreet fallback notice.
|
||||
* C229 — true when the actor asked for `mode=tema` but the sense engine was
|
||||
* unavailable (no provider or no indexed candidate); the page shows the
|
||||
* discreet fallback notice over the literal results.
|
||||
*/
|
||||
themeUnavailable: boolean
|
||||
/** C192 — true when the expansion actually contributed terms. */
|
||||
/** C229 — true when the sense engine actually ran this request. */
|
||||
themeApplied: boolean
|
||||
}
|
||||
|
||||
|
|
@ -211,19 +295,68 @@ export const loadSpeechAcervoPageData = async (
|
|||
payload: Payload,
|
||||
user: CampaignUser,
|
||||
searchParams: Promise<SpeechListSearchParams> | SpeechListSearchParams,
|
||||
expandTheme: SpeechThemeExpansionResolver = expandSpeechSearchTheme,
|
||||
embedQuery: SpeechQueryEmbeddingResolver = embedSpeechQuery,
|
||||
): Promise<SpeechAcervoPageData> => {
|
||||
const rawSearchParams = await searchParams
|
||||
const canonicalUrl = resolveSpeechListUrl(rawSearchParams)
|
||||
const state = canonicalUrl.state
|
||||
|
||||
// C192 — only expand for an actor who may read the catalog (the `find` below
|
||||
// is still the final, fail-closed barrier) and only with a query.
|
||||
const { themeTerms, themeUnavailable } = await resolveSpeechThemeTerms({
|
||||
state,
|
||||
role: user.role,
|
||||
expandTheme,
|
||||
})
|
||||
// C229 — the theme mode is the sense engine: the query is embedded once and
|
||||
// the facet-allowed candidates are ranked over the stored speech-level
|
||||
// vectors. A missing mechanism (no key/provider error) or an empty index
|
||||
// degrades to the literal search below with the discreet notice; the literal
|
||||
// `q` LIKE and the C174 cut-origin lookup never run in the semantic path.
|
||||
const themeRequested = state.mode === 'tema' && Boolean(state.q)
|
||||
const actorCanRead = canReadCommunicationCatalog(user.role)
|
||||
if (themeRequested && actorCanRead) {
|
||||
const queryVector = await embedQuery(state.q ?? '')
|
||||
if (queryVector) {
|
||||
const page = await loadSemanticSpeechPage({
|
||||
payload,
|
||||
user,
|
||||
state,
|
||||
rawSearchParams,
|
||||
queryVector,
|
||||
select: { ...speechListSelect, searchText: true },
|
||||
})
|
||||
if (page) {
|
||||
const speechIds = page.speeches.map((speech) => speech.id)
|
||||
const [cutsBySpeech, filterOptions] = await Promise.all([
|
||||
loadSpeechCutsForSpeeches(payload, user, speechIds),
|
||||
loadSpeechFilterOptions(payload, user),
|
||||
])
|
||||
|
||||
const municipalityIds = new Set<number>()
|
||||
for (const speech of page.speeches) {
|
||||
for (const id of municipalityIdsOfSpeech(speech)) municipalityIds.add(id)
|
||||
}
|
||||
const municipalityLabels = municipalityNameMap(
|
||||
await loadMunicipalityLabelsByIds(payload, [...municipalityIds]),
|
||||
)
|
||||
|
||||
return {
|
||||
rows: page.speeches.map((speech) =>
|
||||
toSpeechListItemViewModel({
|
||||
speech,
|
||||
segments: page.segmentsBySpeech.get(speech.id) ?? [],
|
||||
query: state.q,
|
||||
municipalityLabels,
|
||||
cuts: cutsBySpeech.get(speech.id) ?? [],
|
||||
semanticMatch: true,
|
||||
semanticEvidence: page.evidenceBySpeech.get(speech.id) ?? null,
|
||||
}),
|
||||
),
|
||||
state: page.state,
|
||||
redirectHref: page.redirectHref ?? canonicalUrl.redirectHref,
|
||||
totalDocs: page.totalDocs,
|
||||
totalPages: page.totalPages,
|
||||
filterOptions,
|
||||
themeUnavailable: false,
|
||||
themeApplied: true,
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// C174 (option B) — a speech also shows up when one of its cuts matches the
|
||||
// term by title/description, even if the speech text itself does not.
|
||||
|
|
@ -235,7 +368,7 @@ export const loadSpeechAcervoPageData = async (
|
|||
limit: speechPageSize,
|
||||
page: state.page,
|
||||
sort: '-speechAt',
|
||||
where: buildSpeechListWhereIncludingCutOrigins(state, originSpeechIds, themeTerms),
|
||||
where: buildSpeechListWhereIncludingCutOrigins(state, originSpeechIds),
|
||||
// C174 — the row only needs its normalized text to tell whether it matched
|
||||
// the term itself (option B); the unsearched list does not pay for it.
|
||||
select: state.q ? { ...speechListSelect, searchText: true } : speechListSelect,
|
||||
|
|
@ -261,6 +394,7 @@ export const loadSpeechAcervoPageData = async (
|
|||
await loadMunicipalityLabelsByIds(payload, [...municipalityIds]),
|
||||
)
|
||||
|
||||
const themeUnavailable = themeRequested && actorCanRead
|
||||
return {
|
||||
rows: speeches.map((speech) =>
|
||||
toSpeechListItemViewModel({
|
||||
|
|
@ -269,7 +403,7 @@ export const loadSpeechAcervoPageData = async (
|
|||
query: state.q,
|
||||
municipalityLabels,
|
||||
cuts: cutsBySpeech.get(speech.id) ?? [],
|
||||
themeTerms,
|
||||
literalFallback: themeUnavailable,
|
||||
}),
|
||||
),
|
||||
state: resolvedUrl.state,
|
||||
|
|
@ -278,7 +412,7 @@ export const loadSpeechAcervoPageData = async (
|
|||
totalPages: result.totalPages,
|
||||
filterOptions,
|
||||
themeUnavailable,
|
||||
themeApplied: themeTerms.length > 0,
|
||||
themeApplied: false,
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -299,6 +433,14 @@ const webSpeechListSelect = {
|
|||
thumbnail: true,
|
||||
} as const
|
||||
|
||||
/** C229 — the theme path needs the source text of the evidence windows. */
|
||||
const webSpeechThemeSelect = {
|
||||
...webSpeechListSelect,
|
||||
searchText: true,
|
||||
officialTranscript: true,
|
||||
summary: true,
|
||||
} as const
|
||||
|
||||
const webSpeechDetailSelect = {
|
||||
...webSpeechListSelect,
|
||||
sourceUrl: true,
|
||||
|
|
@ -313,9 +455,9 @@ export type WebSpeechAcervoPageData = {
|
|||
totalDocs: number
|
||||
totalPages: number
|
||||
filterOptions: SpeechFilterOptions
|
||||
/** C216 — true when `mode=tema` was asked but the expansion was unavailable. */
|
||||
/** C229 — true when `mode=tema` was asked but the sense engine was unavailable. */
|
||||
themeUnavailable: boolean
|
||||
/** C216 — true when the expansion actually contributed terms. */
|
||||
/** C229 — true when the sense engine actually ran this request. */
|
||||
themeApplied: boolean
|
||||
}
|
||||
|
||||
|
|
@ -330,7 +472,7 @@ export const loadWebSpeechAcervoPageData = async (
|
|||
payload: Payload,
|
||||
user: CampaignUser,
|
||||
searchParams: Promise<SpeechListSearchParams> | SpeechListSearchParams,
|
||||
expandTheme: SpeechThemeExpansionResolver = expandSpeechSearchTheme,
|
||||
embedQuery: SpeechQueryEmbeddingResolver = embedSpeechQuery,
|
||||
): Promise<WebSpeechAcervoPageData> => {
|
||||
const rawSearchParams = await searchParams
|
||||
// The source is forced (the page dispatches on `source=internet`): the
|
||||
|
|
@ -339,11 +481,45 @@ export const loadWebSpeechAcervoPageData = async (
|
|||
const canonicalUrl = resolveSpeechListUrl(webSourceParams)
|
||||
const state = canonicalUrl.state
|
||||
|
||||
const { themeTerms, themeUnavailable } = await resolveSpeechThemeTerms({
|
||||
state,
|
||||
role: user.role,
|
||||
expandTheme,
|
||||
})
|
||||
// C229 — same sense engine as the Câmara source; the degradation contract is
|
||||
// identical (notice over literal results, never an error).
|
||||
const themeRequested = state.mode === 'tema' && Boolean(state.q)
|
||||
const actorCanRead = canReadCommunicationCatalog(user.role)
|
||||
if (themeRequested && actorCanRead) {
|
||||
const queryVector = await embedQuery(state.q ?? '')
|
||||
if (queryVector) {
|
||||
const page = await loadSemanticSpeechPage({
|
||||
payload,
|
||||
user,
|
||||
state,
|
||||
rawSearchParams: webSourceParams,
|
||||
queryVector,
|
||||
select: webSpeechThemeSelect,
|
||||
})
|
||||
if (page) {
|
||||
const filterOptions = await loadSpeechFilterOptions(payload, user, 'web')
|
||||
|
||||
return {
|
||||
rows: page.speeches.map((speech) =>
|
||||
toWebSpeechListItemViewModel({
|
||||
speech: speech as WebSpeechListRecord,
|
||||
segments: page.segmentsBySpeech.get(speech.id) ?? [],
|
||||
query: state.q,
|
||||
semanticMatch: true,
|
||||
semanticEvidence: page.evidenceBySpeech.get(speech.id) ?? null,
|
||||
}),
|
||||
),
|
||||
state: page.state,
|
||||
redirectHref: page.redirectHref ?? canonicalUrl.redirectHref,
|
||||
totalDocs: page.totalDocs,
|
||||
totalPages: page.totalPages,
|
||||
filterOptions,
|
||||
themeUnavailable: false,
|
||||
themeApplied: true,
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const result = await payload.find({
|
||||
collection: 'speech',
|
||||
|
|
@ -351,7 +527,7 @@ export const loadWebSpeechAcervoPageData = async (
|
|||
limit: speechPageSize,
|
||||
page: state.page,
|
||||
sort: webSpeechSortOrder(state),
|
||||
where: buildSpeechListWhere(state, themeTerms),
|
||||
where: buildSpeechListWhere(state),
|
||||
// The row only needs its normalized text to tell whether it matched the
|
||||
// term itself; the unsearched list does not pay for it.
|
||||
select: state.q ? { ...webSpeechListSelect, searchText: true } : webSpeechListSelect,
|
||||
|
|
@ -368,13 +544,14 @@ export const loadWebSpeechAcervoPageData = async (
|
|||
loadSpeechFilterOptions(payload, user, 'web'),
|
||||
])
|
||||
|
||||
const themeUnavailable = themeRequested && actorCanRead
|
||||
return {
|
||||
rows: speeches.map((speech) =>
|
||||
toWebSpeechListItemViewModel({
|
||||
speech,
|
||||
segments: segmentsBySpeech.get(speech.id) ?? [],
|
||||
query: state.q,
|
||||
themeTerms,
|
||||
literalFallback: themeUnavailable,
|
||||
}),
|
||||
),
|
||||
state: resolvedUrl.state,
|
||||
|
|
@ -383,7 +560,7 @@ export const loadWebSpeechAcervoPageData = async (
|
|||
totalPages: result.totalPages,
|
||||
filterOptions,
|
||||
themeUnavailable,
|
||||
themeApplied: themeTerms.length > 0,
|
||||
themeApplied: false,
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -491,6 +668,7 @@ export const loadSpeechDetailPageData = async (
|
|||
return toSpeechDetailViewModel({
|
||||
speech: speech as SpeechDetailRecord,
|
||||
segments: segments.docs.map((segment) => ({
|
||||
order: segment.order,
|
||||
startSeconds: segment.startSeconds,
|
||||
endSeconds: segment.endSeconds,
|
||||
text: segment.text,
|
||||
|
|
|
|||
225
src/utilities/speech/speechSemanticSearch.ts
Normal file
225
src/utilities/speech/speechSemanticSearch.ts
Normal file
|
|
@ -0,0 +1,225 @@
|
|||
import 'server-only'
|
||||
|
||||
import type { Payload } from 'payload'
|
||||
|
||||
import {
|
||||
pickBestSemanticChunk,
|
||||
rankSemanticHits,
|
||||
sortSemanticHits,
|
||||
speechTextWindows,
|
||||
type SpeechSemanticHit,
|
||||
type SpeechSemanticSort,
|
||||
type SpeechSemanticVector,
|
||||
} from '@/lib/speechSemantic'
|
||||
import type { CampaignUser } from '@/payload-types'
|
||||
import { DEEPINFRA_EMBED_MODEL } from '@/utilities/ai/deepInfraEmbed'
|
||||
import { buildSpeechFacetWhere } from '@/utilities/speech/speechListFilters'
|
||||
import type { SpeechListState } from '@/utilities/speech/speechListUrl'
|
||||
import type {
|
||||
SpeechSegmentRecordWithOrder,
|
||||
SpeechSemanticEvidence,
|
||||
} from '@/utilities/speech/speechViewModels'
|
||||
|
||||
/**
|
||||
* C229 — the semantic path of the acervo list. The candidates are the speeches
|
||||
* the facets allow (never the `q` LIKE branches) and the ranking runs in memory
|
||||
* over the stored `kind: 'speech'` vectors; the page's evidence comes from the
|
||||
* `segment`/`window` rows, loaded only for the current page. Every read goes
|
||||
* through the Payload access layer (`user` + `overrideAccess: false`); the
|
||||
* loader owns the threshold/ordering and the honest degradation.
|
||||
*/
|
||||
|
||||
export type SpeechMeaningSearchResult = {
|
||||
/** Ranked hits (threshold applied), already in the requested order. */
|
||||
hits: SpeechSemanticHit[]
|
||||
/** Facet-allowed speeches before the ranking. */
|
||||
candidates: number
|
||||
/** Candidates with a comparable speech-level vector — zero degrades. */
|
||||
indexedCandidates: number
|
||||
}
|
||||
|
||||
/** Theme mode has no explicit order: relevance is the default (C229/D5). */
|
||||
const semanticSortOf = (state: SpeechListState): SpeechSemanticSort => state.sort ?? 'relevancia'
|
||||
|
||||
const numberVector = (value: unknown): number[] | null => {
|
||||
if (!Array.isArray(value) || value.length === 0) return null
|
||||
if (!value.every((item) => typeof item === 'number' && Number.isFinite(item))) return null
|
||||
return value as number[]
|
||||
}
|
||||
|
||||
/** The index row as this module selects it (never the whole document). */
|
||||
type EmbeddingRow = {
|
||||
id: number
|
||||
speech: number | { id: number } | null
|
||||
kind?: 'speech' | 'segment' | 'window' | null
|
||||
order?: number | null
|
||||
vector?: unknown
|
||||
}
|
||||
|
||||
const speechIdOf = (row: EmbeddingRow): number | null => {
|
||||
const speech = row.speech
|
||||
if (typeof speech === 'number') return speech
|
||||
if (speech && typeof speech === 'object' && typeof speech.id === 'number') return speech.id
|
||||
return null
|
||||
}
|
||||
|
||||
/** Facet-filtered candidates + the speech-level vectors + the in-memory rank. */
|
||||
export const searchSpeechesByMeaning = async (
|
||||
payload: Payload,
|
||||
user: CampaignUser,
|
||||
state: SpeechListState,
|
||||
queryVector: SpeechSemanticVector,
|
||||
): Promise<SpeechMeaningSearchResult> => {
|
||||
const candidates = await payload.find({
|
||||
collection: 'speech',
|
||||
where: { and: buildSpeechFacetWhere(state) },
|
||||
depth: 0,
|
||||
limit: 0,
|
||||
pagination: false,
|
||||
select: { speechAt: true, durationSeconds: true },
|
||||
user,
|
||||
overrideAccess: false,
|
||||
})
|
||||
if (candidates.docs.length === 0) return { hits: [], candidates: 0, indexedCandidates: 0 }
|
||||
|
||||
const embeddings = await payload.find({
|
||||
collection: 'speechEmbedding',
|
||||
where: {
|
||||
and: [
|
||||
{ kind: { equals: 'speech' } },
|
||||
// Only the current model: a stale vector from another embedding space
|
||||
// must never be scored against the query (the loader degrades instead).
|
||||
{ model: { equals: DEEPINFRA_EMBED_MODEL } },
|
||||
{ speech: { in: candidates.docs.map((doc) => doc.id) } },
|
||||
],
|
||||
},
|
||||
depth: 0,
|
||||
limit: 0,
|
||||
pagination: false,
|
||||
select: { speech: true, vector: true },
|
||||
user,
|
||||
overrideAccess: false,
|
||||
})
|
||||
|
||||
const vectorBySpeech = new Map<number, number[]>()
|
||||
for (const row of embeddings.docs) {
|
||||
const speechId = speechIdOf(row)
|
||||
const vector = numberVector(row.vector)
|
||||
if (speechId === null || !vector || vector.length !== queryVector.length) continue
|
||||
vectorBySpeech.set(speechId, vector)
|
||||
}
|
||||
|
||||
const hits = rankSemanticHits(
|
||||
queryVector,
|
||||
candidates.docs.map((doc) => ({
|
||||
id: doc.id,
|
||||
speechAt: doc.speechAt,
|
||||
durationSeconds: doc.durationSeconds ?? null,
|
||||
vector: vectorBySpeech.get(doc.id) ?? null,
|
||||
})),
|
||||
)
|
||||
|
||||
return {
|
||||
hits: sortSemanticHits(hits, semanticSortOf(state)),
|
||||
candidates: candidates.docs.length,
|
||||
indexedCandidates: vectorBySpeech.size,
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolves the real passage of one stored unit. A segment row points at the ASR
|
||||
* segment by `order` (the collection's own ordinal); a window row is recomputed
|
||||
* with the SAME pure splitter the index used, over the same source text. Both
|
||||
* fall back to the first unit when the stored reference is gone (a re-import
|
||||
* that changed the units is the reindex trigger).
|
||||
*/
|
||||
const resolveUnitText = (
|
||||
row: EmbeddingRow,
|
||||
segments: readonly SpeechSegmentRecordWithOrder[],
|
||||
sourceText: string | null,
|
||||
): { text: string; startSeconds: number | null } | null => {
|
||||
if (row.kind === 'segment') {
|
||||
const segment = segments.find((item) => item.order === row.order) ?? segments[0]
|
||||
if (!segment) return null
|
||||
return { text: segment.text, startSeconds: segment.startSeconds }
|
||||
}
|
||||
|
||||
if (row.kind === 'window') {
|
||||
if (!sourceText) return null
|
||||
const windows = speechTextWindows(sourceText)
|
||||
const window =
|
||||
(typeof row.order === 'number' ? windows[row.order] : undefined) ?? windows[0] ?? null
|
||||
return window ? { text: window.text, startSeconds: null } : null
|
||||
}
|
||||
|
||||
return null
|
||||
}
|
||||
|
||||
/**
|
||||
* The evidence of the current page: for each speech, the stored unit closest to
|
||||
* the query. Speeches without a usable unit are simply absent from the map (the
|
||||
* view model falls back to the lexical excerpt).
|
||||
*/
|
||||
export const loadSpeechSemanticEvidence = async (
|
||||
payload: Payload,
|
||||
user: CampaignUser,
|
||||
queryVector: SpeechSemanticVector,
|
||||
speechIds: readonly number[],
|
||||
segmentsBySpeech: ReadonlyMap<number, readonly SpeechSegmentRecordWithOrder[]>,
|
||||
sourceTextBySpeech: ReadonlyMap<number, string | null>,
|
||||
): Promise<Map<number, SpeechSemanticEvidence>> => {
|
||||
const evidence = new Map<number, SpeechSemanticEvidence>()
|
||||
if (speechIds.length === 0) return evidence
|
||||
|
||||
const rows = await payload.find({
|
||||
collection: 'speechEmbedding',
|
||||
where: {
|
||||
and: [
|
||||
{ kind: { in: ['segment', 'window'] } },
|
||||
// Same embedding space only (see `searchSpeechesByMeaning`).
|
||||
{ model: { equals: DEEPINFRA_EMBED_MODEL } },
|
||||
{ speech: { in: [...speechIds] } },
|
||||
],
|
||||
},
|
||||
depth: 0,
|
||||
limit: 0,
|
||||
pagination: false,
|
||||
sort: 'order',
|
||||
select: { speech: true, kind: true, order: true, vector: true },
|
||||
user,
|
||||
overrideAccess: false,
|
||||
})
|
||||
|
||||
const rowsBySpeech = new Map<number, EmbeddingRow[]>()
|
||||
for (const row of rows.docs) {
|
||||
const speechId = speechIdOf(row)
|
||||
if (speechId === null) continue
|
||||
const list = rowsBySpeech.get(speechId) ?? []
|
||||
list.push(row)
|
||||
rowsBySpeech.set(speechId, list)
|
||||
}
|
||||
|
||||
for (const speechId of speechIds) {
|
||||
const speechRows = rowsBySpeech.get(speechId) ?? []
|
||||
const best = pickBestSemanticChunk(
|
||||
queryVector,
|
||||
speechRows.flatMap((row) => {
|
||||
const vector = numberVector(row.vector)
|
||||
if (!vector || vector.length !== queryVector.length) return []
|
||||
return [{ key: row.id, vector }]
|
||||
}),
|
||||
)
|
||||
if (!best) continue
|
||||
|
||||
const row = speechRows.find((candidate) => candidate.id === best.key)
|
||||
if (!row) continue
|
||||
const resolved = resolveUnitText(
|
||||
row,
|
||||
segmentsBySpeech.get(speechId) ?? [],
|
||||
sourceTextBySpeech.get(speechId) ?? null,
|
||||
)
|
||||
if (resolved) evidence.set(speechId, resolved)
|
||||
}
|
||||
|
||||
return evidence
|
||||
}
|
||||
|
|
@ -17,17 +17,12 @@ import type { SpeechScope, SpeechTopic } from '@/lib/speechFacets'
|
|||
import {
|
||||
buildHighlightedExcerpt,
|
||||
pickMatchingSegment,
|
||||
speechSearchTerms,
|
||||
splitHighlightedParts,
|
||||
type SpeechHighlightedExcerpt,
|
||||
type SpeechHighlightPart,
|
||||
} from '@/lib/speechHighlight'
|
||||
import { speechPosterHref, speechPosterTarget } from '@/lib/speechPoster'
|
||||
import {
|
||||
normalizeForSearch,
|
||||
speechMatchesSearchQuery,
|
||||
speechMatchesSearchTerm,
|
||||
} from '@/lib/speechSearch'
|
||||
import { normalizeForSearch, speechMatchesSearchQuery } from '@/lib/speechSearch'
|
||||
import {
|
||||
correctedExcerptOffsetSeconds,
|
||||
excerptOffsetSeconds,
|
||||
|
|
@ -49,6 +44,12 @@ import { speechScopeLabels, speechTopicLabels } from '@/utilities/speech/speechL
|
|||
/** ASR segment shape, owned by the C158 excerpt builder (`lib/speechExcerpt`). */
|
||||
export type SpeechSegmentRecord = SpeechExcerptSegment
|
||||
|
||||
/**
|
||||
* C229 — the segment plus its stored `order`, so the semantic evidence reader
|
||||
* can resolve the index row back to the real passage text.
|
||||
*/
|
||||
export type SpeechSegmentRecordWithOrder = SpeechSegmentRecord & { order: number }
|
||||
|
||||
export type SpeechListRecord = {
|
||||
id: number
|
||||
speechAt: string
|
||||
|
|
@ -106,11 +107,18 @@ export type SpeechListItemViewModel = {
|
|||
*/
|
||||
matchedTextSearch: boolean
|
||||
/**
|
||||
* C192 — the expanded theme term that matched the speech text/keywords, or
|
||||
* null when the search was literal (or the expansion did not surface this
|
||||
* row). The card uses it to label the result and to explain why it appeared.
|
||||
* C229 — the semantic engine surfaced this row in the theme mode and the
|
||||
* excerpt is the real passage closest to the theme. The card uses it to show
|
||||
* the "Tema" seal and the "Trecho mais próximo do tema" block (no highlight,
|
||||
* no score); false in the literal/degraded path.
|
||||
*/
|
||||
themeMatchTerm: string | null
|
||||
semanticMatch: boolean
|
||||
/**
|
||||
* C229 — the actor asked for the theme mode but the engine was down, so this
|
||||
* is a literal result of the degraded path; the card labels it "Termo exato"
|
||||
* (design scene 04) without changing the plain exact search.
|
||||
*/
|
||||
literalFallback: boolean
|
||||
}
|
||||
|
||||
export type SpeechDetailSegmentViewModel = {
|
||||
|
|
@ -193,38 +201,12 @@ const scopeViewModels = (speech: SpeechListRecord) =>
|
|||
(speech.scopes ?? []).map((value) => ({ value, label: speechScopeLabels[value] }))
|
||||
|
||||
/**
|
||||
* C192 — the first expanded theme term that actually surfaced this speech. The
|
||||
* gate is the `speechMatchesSearchTerm` mirror (the same predicate the `where`
|
||||
* runs), so the card never claims a theme the search did not use. The evidence
|
||||
* is the passage that carries the term, or the official keyword when that is
|
||||
* what matched. Returns undefined when no theme term applies to this row.
|
||||
* C229 — the semantic evidence of one row: the real passage closest to the
|
||||
* theme (segment or text window) and its start when the player can seek to it.
|
||||
*/
|
||||
const pickThemeMatch = (
|
||||
speech: SpeechListRecord,
|
||||
segments: readonly SpeechSegmentRecord[],
|
||||
themeTerms: readonly string[],
|
||||
): { term: string; segment?: SpeechSegmentRecord; keyword?: string } | undefined => {
|
||||
for (const term of themeTerms) {
|
||||
const trimmed = term.trim()
|
||||
const normalized = normalizeForSearch(trimmed)
|
||||
if (!normalized || !speechMatchesSearchTerm(speech, trimmed)) continue
|
||||
|
||||
if (normalizeForSearch(speech.searchText ?? '').includes(normalized)) {
|
||||
const segment =
|
||||
segments.find((item) => normalizeForSearch(item.text).includes(normalized)) ??
|
||||
segments.find((item) => {
|
||||
const text = normalizeForSearch(item.text)
|
||||
return speechSearchTerms(trimmed).some((word) => text.includes(word))
|
||||
}) ??
|
||||
segments[0]
|
||||
return { term: trimmed, segment }
|
||||
}
|
||||
|
||||
const lowered = trimmed.toLowerCase()
|
||||
const keyword = (speech.keywords ?? []).find((item) => item.toLowerCase().includes(lowered))
|
||||
if (keyword) return { term: trimmed, keyword }
|
||||
}
|
||||
return undefined
|
||||
export type SpeechSemanticEvidence = {
|
||||
text: string
|
||||
startSeconds: number | null
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -234,11 +216,13 @@ const pickThemeMatch = (
|
|||
*/
|
||||
const withSeekQuery = (
|
||||
href: string,
|
||||
segment: SpeechSegmentRecord | undefined,
|
||||
startSeconds: number | null | undefined,
|
||||
query: string | undefined,
|
||||
): string => {
|
||||
const params = new URLSearchParams()
|
||||
if (segment) params.set('t', String(Math.max(0, Math.floor(segment.startSeconds))))
|
||||
if (typeof startSeconds === 'number' && Number.isFinite(startSeconds)) {
|
||||
params.set('t', String(Math.max(0, Math.floor(startSeconds))))
|
||||
}
|
||||
if (query) params.set('q', query)
|
||||
const queryString = params.toString()
|
||||
return queryString ? `${href}?${queryString}` : href
|
||||
|
|
@ -246,9 +230,9 @@ const withSeekQuery = (
|
|||
|
||||
export const buildWatchHref = (
|
||||
speechId: number,
|
||||
segment: SpeechSegmentRecord | undefined,
|
||||
startSeconds: number | null | undefined,
|
||||
query: string | undefined,
|
||||
): string => withSeekQuery(`${CAMPAIGN_COMMUNICATION_ACERVO}/${speechId}`, segment, query)
|
||||
): string => withSeekQuery(`${CAMPAIGN_COMMUNICATION_ACERVO}/${speechId}`, startSeconds, query)
|
||||
|
||||
type SpeechCuts = readonly SpeechCutSummaryViewModel[]
|
||||
|
||||
|
|
@ -261,7 +245,9 @@ export const toSpeechListItemViewModel = ({
|
|||
query,
|
||||
municipalityLabels,
|
||||
cuts = [],
|
||||
themeTerms = [],
|
||||
semanticMatch = false,
|
||||
literalFallback = false,
|
||||
semanticEvidence = null,
|
||||
}: {
|
||||
speech: SpeechListRecord
|
||||
segments: readonly SpeechSegmentRecord[]
|
||||
|
|
@ -269,8 +255,12 @@ export const toSpeechListItemViewModel = ({
|
|||
municipalityLabels: ReadonlyMap<number, string>
|
||||
/** C174 — the cuts of this speech, nested in the result card. */
|
||||
cuts?: SpeechCuts
|
||||
/** C192 — expanded theme terms; empty in the literal search. */
|
||||
themeTerms?: readonly string[]
|
||||
/** C229 — the sense engine surfaced this row (a hit, by construction). */
|
||||
semanticMatch?: boolean
|
||||
/** C229 — literal result of the degraded theme path (design scene 04). */
|
||||
literalFallback?: boolean
|
||||
/** C229 — the passage closest to the theme; null when the row has no unit. */
|
||||
semanticEvidence?: SpeechSemanticEvidence | null
|
||||
}): SpeechListItemViewModel => {
|
||||
const matchedSegment = pickMatchingSegment(segments, query)
|
||||
const q = query?.trim()
|
||||
|
|
@ -280,25 +270,18 @@ export const toSpeechListItemViewModel = ({
|
|||
Boolean(normalizedQuery) &&
|
||||
(speech.keywords ?? []).some((keyword) => normalizeForSearch(keyword).includes(normalizedQuery))
|
||||
|
||||
// C192 — the theme evidence wins the excerpt so the card's "Por que apareceu"
|
||||
// block shows the passage that actually matched, even when the speech also
|
||||
// contains the literal query.
|
||||
const themeMatch = pickThemeMatch(speech, segments, themeTerms)
|
||||
|
||||
// C229 — the semantic evidence wins the excerpt so the theme card shows the
|
||||
// real passage closest to the sense searched, without any highlight (the
|
||||
// theme engine never matched a term) and without a score.
|
||||
const excerptSource =
|
||||
themeMatch?.segment?.text ??
|
||||
themeMatch?.keyword ??
|
||||
semanticEvidence?.text ??
|
||||
matchedSegment?.text ??
|
||||
segments[0]?.text ??
|
||||
speech.summary ??
|
||||
speech.officialTranscript ??
|
||||
''
|
||||
|
||||
// C192 — the theme passage highlights the whole phrase as one band (the
|
||||
// per-term split would drop the "à" and break it into pieces).
|
||||
const excerpt = themeMatch
|
||||
? buildHighlightedExcerpt(excerptSource, themeMatch.term, { phrase: true })
|
||||
: buildHighlightedExcerpt(excerptSource, q ?? '')
|
||||
const excerpt = buildHighlightedExcerpt(excerptSource, semanticMatch ? '' : (q ?? ''))
|
||||
|
||||
return {
|
||||
id: speech.id,
|
||||
|
|
@ -307,23 +290,28 @@ export const toSpeechListItemViewModel = ({
|
|||
durationLabel: formatSpeechDuration(speech.durationSeconds),
|
||||
presidingOfficer: speech.presidingOfficer ?? null,
|
||||
excerpt,
|
||||
matchKind: matchedSegment
|
||||
? 'segment'
|
||||
: keywordMatch
|
||||
? 'keyword'
|
||||
: themeMatch
|
||||
? 'theme'
|
||||
matchKind: semanticMatch
|
||||
? 'theme'
|
||||
: matchedSegment
|
||||
? 'segment'
|
||||
: keywordMatch
|
||||
? 'keyword'
|
||||
: 'fallback',
|
||||
topics: topicViewModels(speech),
|
||||
scopes: scopeViewModels(speech),
|
||||
keywords: speech.keywords ?? [],
|
||||
municipalities: municipalityViewModels(speech, municipalityLabels),
|
||||
thumbnailUrl: thumbnailUrlOf(speech),
|
||||
watchHref: buildWatchHref(speech.id, matchedSegment, q),
|
||||
watchHref: buildWatchHref(
|
||||
speech.id,
|
||||
semanticEvidence?.startSeconds ?? matchedSegment?.startSeconds,
|
||||
q,
|
||||
),
|
||||
officialTextUrl: speech.officialTextUrl ?? null,
|
||||
cuts: [...cuts],
|
||||
matchedTextSearch: speechMatchesSearchQuery(speech, query),
|
||||
themeMatchTerm: themeMatch?.term ?? null,
|
||||
semanticMatch,
|
||||
literalFallback,
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -403,7 +391,7 @@ export type WebSpeechListRecord = {
|
|||
scopes?: SpeechScope[] | null
|
||||
/** Upload relation at depth 0 (id) or depth 1 (object); presence drives the cover. */
|
||||
thumbnail?: number | { id: number } | null
|
||||
/** Normalized search text; lets the row mirror the Câmara theme evidence. */
|
||||
/** Normalized search text; lets the row mirror the Câmara literal match. */
|
||||
searchText?: string | null
|
||||
}
|
||||
|
||||
|
|
@ -430,6 +418,12 @@ export type WebSpeechListItemViewModel = {
|
|||
/** Detail link that seeks the player to the matching segment (`?t=`) and
|
||||
* keeps the search term for the transcript highlight (`?q=`). */
|
||||
watchHref: string
|
||||
/** C229 — the semantic engine surfaced this row; same contract as the Câmara. */
|
||||
semanticMatch: boolean
|
||||
/** C229 — literal result of the degraded theme path (design scene 04). */
|
||||
literalFallback: boolean
|
||||
/** C229 — the row also contains the literal query; the card shows "Termo exato". */
|
||||
matchedTextSearch: boolean
|
||||
}
|
||||
|
||||
export type WebSpeechDetailViewModel = {
|
||||
|
|
@ -464,38 +458,36 @@ const webSpeechTitle = (speech: WebSpeechListRecord): string =>
|
|||
|
||||
const buildWebSpeechWatchHref = (
|
||||
speechId: number,
|
||||
segment: SpeechSegmentRecord | undefined,
|
||||
startSeconds: number | null | undefined,
|
||||
query: string | undefined,
|
||||
): string => withSeekQuery(campaignInternetSpeechHref(speechId), segment, query)
|
||||
): string => withSeekQuery(campaignInternetSpeechHref(speechId), startSeconds, query)
|
||||
|
||||
/**
|
||||
* List item of one web speech: platform chip, publication date + duration, the
|
||||
* matching excerpt (theme evidence preferred, like the Câmara) and the private
|
||||
* cover href. The cuts/VOD of the Câmara row have no counterpart here.
|
||||
* matching excerpt (C229 semantic evidence preferred, like the Câmara) and the
|
||||
* private cover href. The cuts/VOD of the Câmara row have no counterpart here.
|
||||
*/
|
||||
export const toWebSpeechListItemViewModel = ({
|
||||
speech,
|
||||
segments,
|
||||
query,
|
||||
themeTerms = [],
|
||||
semanticMatch = false,
|
||||
literalFallback = false,
|
||||
semanticEvidence = null,
|
||||
}: {
|
||||
speech: WebSpeechListRecord
|
||||
segments: readonly SpeechSegmentRecord[]
|
||||
query?: string
|
||||
/** C216 — expanded theme terms; empty in the literal search. */
|
||||
themeTerms?: readonly string[]
|
||||
/** C229 — the sense engine surfaced this row (a hit, by construction). */
|
||||
semanticMatch?: boolean
|
||||
/** C229 — literal result of the degraded theme path (design scene 04). */
|
||||
literalFallback?: boolean
|
||||
/** C229 — the passage closest to the theme; null when the row has no unit. */
|
||||
semanticEvidence?: SpeechSemanticEvidence | null
|
||||
}): WebSpeechListItemViewModel => {
|
||||
const matchedSegment = pickMatchingSegment(segments, query)
|
||||
const q = query?.trim()
|
||||
// Same evidence rule as the Câmara VM: a theme passage wins the excerpt so
|
||||
// the card shows why the row appeared.
|
||||
const themeMatch = pickThemeMatch(speech, segments, themeTerms)
|
||||
const excerptSource =
|
||||
themeMatch?.segment?.text ??
|
||||
themeMatch?.keyword ??
|
||||
matchedSegment?.text ??
|
||||
segments[0]?.text ??
|
||||
''
|
||||
const excerptSource = semanticEvidence?.text ?? matchedSegment?.text ?? segments[0]?.text ?? ''
|
||||
|
||||
return {
|
||||
id: speech.id,
|
||||
|
|
@ -503,14 +495,19 @@ export const toWebSpeechListItemViewModel = ({
|
|||
platform: webPlatformViewModel(speech.platform),
|
||||
dateLabel: formatSpeechDate(speech.speechAt),
|
||||
durationLabel: formatSpeechDuration(speech.durationSeconds),
|
||||
excerpt: themeMatch
|
||||
? buildHighlightedExcerpt(excerptSource, themeMatch.term, { phrase: true })
|
||||
: buildHighlightedExcerpt(excerptSource, q ?? ''),
|
||||
excerpt: buildHighlightedExcerpt(excerptSource, semanticMatch ? '' : (q ?? '')),
|
||||
topics: topicViewModels(speech),
|
||||
scopes: scopeViewModels(speech),
|
||||
thumbnailUrl:
|
||||
relationshipId(speech.thumbnail) === null ? null : campaignInternetSpeechCoverHref(speech.id),
|
||||
watchHref: buildWebSpeechWatchHref(speech.id, matchedSegment ?? themeMatch?.segment, q),
|
||||
watchHref: buildWebSpeechWatchHref(
|
||||
speech.id,
|
||||
semanticEvidence?.startSeconds ?? matchedSegment?.startSeconds,
|
||||
q,
|
||||
),
|
||||
semanticMatch,
|
||||
literalFallback,
|
||||
matchedTextSearch: speechMatchesSearchQuery(speech, query),
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -339,7 +339,7 @@ test.describe('communication vertical (C154/C162)', () => {
|
|||
expect(filteredHtml).toContain('Limpar busca e filtros')
|
||||
})
|
||||
|
||||
test('theme mode degrades honestly to the exact search without the key (C192)', async ({
|
||||
test('theme mode degrades honestly to the exact search without the key (C229)', async ({
|
||||
campaign,
|
||||
campaignRequest,
|
||||
}) => {
|
||||
|
|
@ -349,8 +349,9 @@ test.describe('communication vertical (C154/C162)', () => {
|
|||
const user = await campaign.fixtures.createCampaignUser('communicator')
|
||||
const request = await campaignRequest(user, user.password)
|
||||
|
||||
// The e2e environment blanks DEEPSEEK_API_KEY (playwright.config.ts), so the
|
||||
// expansion is unavailable and the page must fall back with the notice.
|
||||
// The e2e environment blanks DEEPINFRA_API_KEY (playwright.config.ts), so
|
||||
// the sense engine is unavailable and the page must fall back with the
|
||||
// notice.
|
||||
const themed = await request.get(`/campanha/comunicacao/acervo?q=${marker}&mode=tema`)
|
||||
expect(themed.status()).toBe(200)
|
||||
const themedHtml = rendered(await themed.text())
|
||||
|
|
@ -358,22 +359,49 @@ test.describe('communication vertical (C154/C162)', () => {
|
|||
expect(themedHtml).toContain('Termo exato')
|
||||
expect(themedHtml).toContain('Por tema')
|
||||
expect(themedHtml).toContain('data-testid="speech-theme-fallback"')
|
||||
expect(themedHtml).toContain('A busca por tema está indisponível agora.')
|
||||
expect(themedHtml).toContain('A busca por sentido está indisponível agora.')
|
||||
expect(themedHtml).toContain('Tentar por tema novamente')
|
||||
expect(themedHtml).toContain('Resultados por termo exato')
|
||||
expect(themedHtml).toContain('data-testid="speech-theme-fallback"')
|
||||
// The degraded literal card carries the term seal (design scene 04).
|
||||
expect(themedHtml).toContain('data-testid="speech-literal-badge"')
|
||||
// The exact results still render and keep the highlighted term.
|
||||
expect(themedHtml).toContain(marker)
|
||||
expect(themedHtml).toContain('<mark')
|
||||
// No theme badge is fabricated.
|
||||
expect(themedHtml).not.toContain('Por que apareceu')
|
||||
// No semantic evidence is fabricated.
|
||||
expect(themedHtml).not.toContain('Trecho mais próximo do tema')
|
||||
|
||||
// The default mode stays untouched: no mode param means no notice.
|
||||
const exact = await request.get(`/campanha/comunicacao/acervo?q=${marker}`)
|
||||
expect(exact.status()).toBe(200)
|
||||
const exactHtml = rendered(await exact.text())
|
||||
expect(exactHtml).toContain(marker)
|
||||
expect(exactHtml).not.toContain('A busca por tema está indisponível agora.')
|
||||
expect(exactHtml).not.toContain('A busca por sentido está indisponível agora.')
|
||||
})
|
||||
|
||||
test('the web theme sort canonicalizes without a redirect loop (C229)', async ({
|
||||
campaign,
|
||||
campaignRequest,
|
||||
}) => {
|
||||
const marker = campaign.fixtures.value('temaordem')
|
||||
const user = await campaign.fixtures.createCampaignUser('communicator')
|
||||
const request = await campaignRequest(user, user.password)
|
||||
|
||||
const recentes = await request.get(
|
||||
`/campanha/comunicacao/acervo?source=internet&q=${marker}&mode=tema&sort=recentes`,
|
||||
)
|
||||
expect(recentes.status()).toBe(200)
|
||||
const recentesHtml = rendered(await recentes.text())
|
||||
// The engine is down in e2e (no key): the ordering control honestly shows
|
||||
// the literal vocabulary and the explicit order survives canonicalization.
|
||||
expect(recentesHtml).toContain('Mais recentes')
|
||||
expect(recentesHtml).not.toContain('Mais relevantes')
|
||||
expect(recentesHtml).toContain('A busca por sentido está indisponível agora.')
|
||||
|
||||
// `relevancia` is the theme default: canonicalizes away, no loop.
|
||||
const relevance = await request.get(
|
||||
`/campanha/comunicacao/acervo?source=internet&q=${marker}&mode=tema&sort=relevancia`,
|
||||
)
|
||||
expect(relevance.status()).toBe(200)
|
||||
})
|
||||
|
||||
test('advisor and leader are redirected away from the acervo', async ({
|
||||
|
|
|
|||
|
|
@ -6,9 +6,12 @@ import type { Payload } from 'payload'
|
|||
import { getPayload } from 'payload'
|
||||
import { afterAll, beforeAll, describe, expect, it } from 'vitest'
|
||||
|
||||
import type { SpeechTopic } from '@/lib/speechFacets'
|
||||
import { matchMunicipalityMentions } from '@/lib/speechGazetteer'
|
||||
import { speechPosterHref } from '@/lib/speechPoster'
|
||||
import config from '@/payload.config'
|
||||
import { DEEPINFRA_EMBED_MODEL } from '@/utilities/ai/deepInfraEmbed'
|
||||
import { indexSpeechSources } from '@/utilities/speech/speechEmbeddingIndex'
|
||||
import { upsertSpeechBundle, type SpeechImportBundle } from '@/utilities/speech/speechImport'
|
||||
import {
|
||||
loadSpeechAcervoPageData,
|
||||
|
|
@ -65,6 +68,42 @@ const createSpeech = async (overrides: Partial<SpeechImportBundle> = {}): Promis
|
|||
return speech.id
|
||||
}
|
||||
|
||||
/**
|
||||
* C229 — writes one index row directly (the CLI has its own unit coverage);
|
||||
* the vectors are tiny and hand-picked so the ranking assertions are exact.
|
||||
*/
|
||||
const seedSpeechEmbedding = async (
|
||||
speechId: number,
|
||||
kind: 'speech' | 'segment' | 'window',
|
||||
order: number | null,
|
||||
vector: number[],
|
||||
): Promise<void> => {
|
||||
await payload.create({
|
||||
collection: 'speechEmbedding',
|
||||
data: {
|
||||
speech: speechId,
|
||||
kind,
|
||||
order,
|
||||
model: DEEPINFRA_EMBED_MODEL,
|
||||
dimensions: vector.length,
|
||||
contentHash: `${kind}-${order ?? 'root'}-${vector.join(',')}`,
|
||||
vector,
|
||||
},
|
||||
overrideAccess: true,
|
||||
})
|
||||
}
|
||||
|
||||
/** Facets of a test speech with the topic overridden (the facet boundary test). */
|
||||
const facetsWithTopics = (topics: SpeechTopic[]): NonNullable<SpeechImportBundle['facets']> => ({
|
||||
topics,
|
||||
scopes: ['bahia'],
|
||||
municipalities: [],
|
||||
people: [],
|
||||
programs: [],
|
||||
projects: [],
|
||||
classifiedBy: 'gazetteer',
|
||||
})
|
||||
|
||||
/**
|
||||
* C172 — the detail loader asks YouTube for the session video's start by
|
||||
* default; the fixtures keep the suite offline by injecting a resolver.
|
||||
|
|
@ -348,42 +387,75 @@ describe('speech acervo (C154)', () => {
|
|||
).rejects.toBeInstanceOf(SpeechNotFoundError)
|
||||
})
|
||||
|
||||
it('finds a speech by expanded theme terms and explains the match (C192)', async () => {
|
||||
it('ranks by sense with the closest real passage and no highlight (C229)', async () => {
|
||||
const runId = randomUUID().slice(0, 8)
|
||||
const marker = `tema${runId}`
|
||||
const id = await createSpeech({
|
||||
const closer = await createSpeech({
|
||||
speechAt: '2020-01-01T10:00',
|
||||
segments: [
|
||||
{
|
||||
startSeconds: 0,
|
||||
endSeconds: 3,
|
||||
text: `Garantir o atendimento universal ${marker} na rede pública`,
|
||||
},
|
||||
{ startSeconds: 10, endSeconds: 13, text: 'Primeiro trecho da fala' },
|
||||
{ startSeconds: 40, endSeconds: 43, text: `O embate com a oposição ${runId}` },
|
||||
],
|
||||
})
|
||||
const newer = await createSpeech({
|
||||
speechAt: '2026-01-01T10:00',
|
||||
segments: [{ startSeconds: 0, endSeconds: 3, text: 'Um assunto distante' }],
|
||||
})
|
||||
await seedSpeechEmbedding(closer, 'speech', null, [1, 0])
|
||||
await seedSpeechEmbedding(newer, 'speech', null, [0.9, 0.435])
|
||||
await seedSpeechEmbedding(closer, 'segment', 2, [1, 0])
|
||||
await seedSpeechEmbedding(closer, 'segment', 1, [0, 1])
|
||||
await seedSpeechEmbedding(newer, 'segment', 1, [0.9, 0.435])
|
||||
|
||||
const { coordinator } = await createUsers()
|
||||
const expansionCalls: string[] = []
|
||||
const embedCalls: string[] = []
|
||||
const data = await loadSpeechAcervoPageData(
|
||||
payload,
|
||||
coordinator,
|
||||
{ q: 'defesa do SUS', mode: 'tema' },
|
||||
async (theme) => {
|
||||
expansionCalls.push(theme)
|
||||
return { terms: [`atendimento universal ${marker}`] }
|
||||
{ q: 'combate à oposição', mode: 'tema' },
|
||||
async (query) => {
|
||||
embedCalls.push(query)
|
||||
return [1, 0]
|
||||
},
|
||||
)
|
||||
|
||||
expect(expansionCalls).toEqual(['defesa do SUS'])
|
||||
expect(embedCalls).toEqual(['combate à oposição'])
|
||||
expect(data.themeUnavailable).toBe(false)
|
||||
expect(data.themeApplied).toBe(true)
|
||||
const row = data.rows.find((item) => item.id === id)
|
||||
expect(row?.matchKind).toBe('theme')
|
||||
expect(row?.themeMatchTerm).toBe(`atendimento universal ${marker}`)
|
||||
// Proximity beats the date: the older speech outranks the newer one.
|
||||
const ids = data.rows.map((row) => row.id)
|
||||
expect(ids.indexOf(closer)).toBeLessThan(ids.indexOf(newer))
|
||||
|
||||
const row = data.rows.find((item) => item.id === closer)
|
||||
expect(row?.semanticMatch).toBe(true)
|
||||
expect(row?.matchedTextSearch).toBe(false)
|
||||
expect(row?.excerpt.parts.some((part) => part.highlighted)).toBe(true)
|
||||
expect(row?.matchKind).toBe('theme')
|
||||
// The evidence is the real segment closest to the query, without highlight.
|
||||
expect(row?.excerpt.parts.map((part) => part.text).join('')).toContain(
|
||||
`embate com a oposição ${runId}`,
|
||||
)
|
||||
expect(row?.excerpt.parts.some((part) => part.highlighted)).toBe(false)
|
||||
expect(row?.watchHref).toContain('t=40')
|
||||
})
|
||||
|
||||
it('degrades to the literal search when the expansion is unavailable (C192)', async () => {
|
||||
it('returns the honest empty when nothing passes the cut-off (C229)', async () => {
|
||||
const runId = randomUUID().slice(0, 8)
|
||||
const id = await createSpeech({ speechAt: '1985-01-01T10:00', year: 1985 })
|
||||
await seedSpeechEmbedding(id, 'speech', null, [0, 1])
|
||||
|
||||
const { coordinator } = await createUsers()
|
||||
const data = await loadSpeechAcervoPageData(
|
||||
payload,
|
||||
coordinator,
|
||||
{ q: `tema distante ${runId}`, mode: 'tema', year: '1985' },
|
||||
async () => [1, 0],
|
||||
)
|
||||
|
||||
expect(data.rows).toEqual([])
|
||||
expect(data.themeApplied).toBe(true)
|
||||
expect(data.themeUnavailable).toBe(false)
|
||||
})
|
||||
|
||||
it('degrades to the literal search when the embedder is unavailable (C229)', async () => {
|
||||
const runId = randomUUID().slice(0, 8)
|
||||
const marker = `literal${runId}`
|
||||
const id = await createSpeech({
|
||||
|
|
@ -401,17 +473,198 @@ describe('speech acervo (C154)', () => {
|
|||
expect(data.themeUnavailable).toBe(true)
|
||||
expect(data.themeApplied).toBe(false)
|
||||
const row = data.rows.find((item) => item.id === id)
|
||||
expect(row?.themeMatchTerm).toBeNull()
|
||||
expect(row?.semanticMatch).toBe(false)
|
||||
expect(row?.matchedTextSearch).toBe(true)
|
||||
})
|
||||
|
||||
it('never expands for an actor outside the catalog gate (C192)', async () => {
|
||||
it('degrades when the candidates have no index rows (C229)', async () => {
|
||||
const id = await createSpeech({ speechAt: '1984-01-01T10:00', year: 1984 })
|
||||
|
||||
const { coordinator } = await createUsers()
|
||||
const data = await loadSpeechAcervoPageData(
|
||||
payload,
|
||||
coordinator,
|
||||
{ q: 'qualquer tema', mode: 'tema', year: '1984' },
|
||||
async () => [1, 0],
|
||||
)
|
||||
|
||||
expect(data.themeUnavailable).toBe(true)
|
||||
expect(data.themeApplied).toBe(false)
|
||||
expect(data.rows.every((row) => row.id !== id)).toBe(true)
|
||||
})
|
||||
|
||||
it('keeps the facets as the candidate boundary (C229)', async () => {
|
||||
const runId = randomUUID().slice(0, 8)
|
||||
const saude = await createSpeech({
|
||||
speechAt: '2024-01-01T10:00',
|
||||
facets: facetsWithTopics(['saude']),
|
||||
segments: [{ startSeconds: 0, endSeconds: 3, text: `Saúde ${runId}` }],
|
||||
})
|
||||
const educacao = await createSpeech({
|
||||
speechAt: '2024-01-02T10:00',
|
||||
facets: facetsWithTopics(['educacao']),
|
||||
segments: [{ startSeconds: 0, endSeconds: 3, text: `Educação ${runId}` }],
|
||||
})
|
||||
await seedSpeechEmbedding(saude, 'speech', null, [1, 0])
|
||||
await seedSpeechEmbedding(educacao, 'speech', null, [1, 0])
|
||||
|
||||
const { coordinator } = await createUsers()
|
||||
const data = await loadSpeechAcervoPageData(
|
||||
payload,
|
||||
coordinator,
|
||||
{ q: 'tema qualquer', mode: 'tema', topic: 'saude' },
|
||||
async () => [1, 0],
|
||||
)
|
||||
|
||||
const ids = data.rows.map((row) => row.id)
|
||||
expect(ids).toContain(saude)
|
||||
expect(ids).not.toContain(educacao)
|
||||
})
|
||||
|
||||
it('ranks by relevance over the date in the theme mode (C229)', async () => {
|
||||
const runId = randomUUID().slice(0, 8)
|
||||
const older = await createSpeech({
|
||||
speechAt: '2020-01-01T10:00',
|
||||
segments: [{ startSeconds: 0, endSeconds: 3, text: `Fala ${runId}` }],
|
||||
})
|
||||
const newer = await createSpeech({
|
||||
speechAt: '2026-01-01T10:00',
|
||||
segments: [{ startSeconds: 0, endSeconds: 3, text: `Fala ${runId}` }],
|
||||
})
|
||||
await seedSpeechEmbedding(older, 'speech', null, [1, 0])
|
||||
await seedSpeechEmbedding(newer, 'speech', null, [0.9, 0.435])
|
||||
|
||||
const { coordinator } = await createUsers()
|
||||
const data = await loadSpeechAcervoPageData(
|
||||
payload,
|
||||
coordinator,
|
||||
{ q: `tema ${runId}`, mode: 'tema' },
|
||||
async () => [1, 0],
|
||||
)
|
||||
|
||||
// Relevance first: the older speech outranks the newer one.
|
||||
const ids = data.rows.map((row) => row.id)
|
||||
expect(ids.indexOf(older)).toBeLessThan(ids.indexOf(newer))
|
||||
// The Câmara contract has no sort: the dated state never leaks through.
|
||||
expect(data.state.sort).toBeUndefined()
|
||||
})
|
||||
|
||||
it('indexes a speech end-to-end and skips it on the second run (C229)', async () => {
|
||||
const runId = randomUUID().slice(0, 8)
|
||||
const id = await createSpeech({
|
||||
speechAt: '1986-01-01T10:00',
|
||||
year: 1986,
|
||||
segments: [
|
||||
{ startSeconds: 0, endSeconds: 3, text: `Primeiro trecho ${runId}` },
|
||||
{ startSeconds: 3, endSeconds: 6, text: `Segundo trecho ${runId}` },
|
||||
],
|
||||
})
|
||||
const source = {
|
||||
id,
|
||||
segments: [
|
||||
{ order: 1, startSeconds: 0, text: `Primeiro trecho ${runId}` },
|
||||
{ order: 2, startSeconds: 3, text: `Segundo trecho ${runId}` },
|
||||
],
|
||||
}
|
||||
const embedCalls: string[][] = []
|
||||
const embedTexts = async (texts: readonly string[]) => {
|
||||
embedCalls.push([...texts])
|
||||
return {
|
||||
vectors: texts.map((_, index) => (index === 0 ? [1, 0] : [0.6, 0.8])),
|
||||
promptTokens: 10,
|
||||
}
|
||||
}
|
||||
|
||||
const first = await indexSpeechSources(payload, [source], { embedTexts })
|
||||
expect(first).toMatchObject({ indexed: 1, failed: 0, units: 2, upToDate: 0 })
|
||||
expect(embedCalls).toEqual([[`Primeiro trecho ${runId}`, `Segundo trecho ${runId}`]])
|
||||
|
||||
const { coordinator } = await createUsers()
|
||||
const data = await loadSpeechAcervoPageData(
|
||||
payload,
|
||||
coordinator,
|
||||
{ q: `tema ${runId}`, mode: 'tema', year: '1986' },
|
||||
async () => [1, 0],
|
||||
)
|
||||
const row = data.rows.find((item) => item.id === id)
|
||||
expect(row?.semanticMatch).toBe(true)
|
||||
expect(row?.excerpt.parts.map((part) => part.text).join('')).toContain(
|
||||
`Primeiro trecho ${runId}`,
|
||||
)
|
||||
|
||||
const second = await indexSpeechSources(payload, [source], { embedTexts })
|
||||
expect(second).toMatchObject({ indexed: 0, failed: 0, upToDate: 1 })
|
||||
expect(embedCalls).toHaveLength(1)
|
||||
|
||||
// A stored order that no longer exists falls back to the FIRST real segment
|
||||
// (never a shifted positional guess).
|
||||
await seedSpeechEmbedding(id, 'segment', 999, [0, 1])
|
||||
const fallback = await loadSpeechAcervoPageData(
|
||||
payload,
|
||||
coordinator,
|
||||
{ q: `tema ${runId}`, mode: 'tema', year: '1986' },
|
||||
async () => [0, 1],
|
||||
)
|
||||
expect(
|
||||
fallback.rows
|
||||
.find((item) => item.id === id)
|
||||
?.excerpt.parts.map((part) => part.text)
|
||||
.join(''),
|
||||
).toContain(`Primeiro trecho ${runId}`)
|
||||
})
|
||||
|
||||
it('cascades the index rows when the speech is deleted (C229)', async () => {
|
||||
const id = await createSpeech()
|
||||
await seedSpeechEmbedding(id, 'speech', null, [1, 0])
|
||||
|
||||
await payload.delete({ collection: 'speech', id, depth: 0, overrideAccess: true })
|
||||
|
||||
const found = await payload.find({
|
||||
collection: 'speechEmbedding',
|
||||
where: { speech: { equals: id } },
|
||||
depth: 0,
|
||||
limit: 1,
|
||||
overrideAccess: true,
|
||||
})
|
||||
expect(found.totalDocs).toBe(0)
|
||||
})
|
||||
|
||||
it('ignores vectors from another embedding model (C229)', async () => {
|
||||
const id = await createSpeech({ speechAt: '1983-01-01T10:00', year: 1983 })
|
||||
await payload.create({
|
||||
collection: 'speechEmbedding',
|
||||
data: {
|
||||
speech: id,
|
||||
kind: 'speech',
|
||||
model: 'outro-modelo',
|
||||
dimensions: 2,
|
||||
contentHash: 'outro-modelo',
|
||||
vector: [1, 0],
|
||||
},
|
||||
overrideAccess: true,
|
||||
})
|
||||
|
||||
const { coordinator } = await createUsers()
|
||||
const data = await loadSpeechAcervoPageData(
|
||||
payload,
|
||||
coordinator,
|
||||
{ q: 'qualquer tema', mode: 'tema', year: '1983' },
|
||||
async () => [1, 0],
|
||||
)
|
||||
|
||||
// The foreign vector is never scored against a bge-m3 query: the loader
|
||||
// degrades instead of ranking nonsense.
|
||||
expect(data.themeUnavailable).toBe(true)
|
||||
expect(data.rows).toEqual([])
|
||||
})
|
||||
|
||||
it('never calls the sense engine for an actor outside the catalog gate (C229)', async () => {
|
||||
const { advisor } = await createUsers()
|
||||
let called = false
|
||||
await expect(
|
||||
loadSpeechAcervoPageData(payload, advisor, { q: 'SUS', mode: 'tema' }, async () => {
|
||||
called = true
|
||||
return { terms: ['saúde'] }
|
||||
return [1, 0]
|
||||
}),
|
||||
).rejects.toThrow(/permissão/i)
|
||||
expect(called).toBe(false)
|
||||
|
|
|
|||
|
|
@ -8,6 +8,7 @@ import { afterAll, beforeAll, describe, expect, it } from 'vitest'
|
|||
import { matchMunicipalityMentions } from '@/lib/speechGazetteer'
|
||||
import { INTERNET_SPEECH_MEDIA_SLUG } from '@/lib/webSpeech'
|
||||
import config from '@/payload.config'
|
||||
import { DEEPINFRA_EMBED_MODEL } from '@/utilities/ai/deepInfraEmbed'
|
||||
import { upsertSpeechBundle, type SpeechImportBundle } from '@/utilities/speech/speechImport'
|
||||
import {
|
||||
loadSpeechAcervoPageData,
|
||||
|
|
@ -95,6 +96,28 @@ const createWebSpeech = async (overrides: Partial<SpeechImportBundle> = {}): Pro
|
|||
return speech.id
|
||||
}
|
||||
|
||||
/** C229 — one hand-picked index row for the web source (vectors are tiny on purpose). */
|
||||
const seedSpeechEmbedding = async (
|
||||
speechId: number,
|
||||
kind: 'speech' | 'segment' | 'window',
|
||||
order: number | null,
|
||||
vector: number[],
|
||||
): Promise<void> => {
|
||||
await payload.create({
|
||||
collection: 'speechEmbedding',
|
||||
data: {
|
||||
speech: speechId,
|
||||
kind,
|
||||
order,
|
||||
model: DEEPINFRA_EMBED_MODEL,
|
||||
dimensions: vector.length,
|
||||
contentHash: `${kind}-${order ?? 'root'}-${vector.join(',')}`,
|
||||
vector,
|
||||
},
|
||||
overrideAccess: true,
|
||||
})
|
||||
}
|
||||
|
||||
const createCamaraSpeech = async (overrides: Partial<SpeechImportBundle> = {}): Promise<number> => {
|
||||
const bundle = speechBundleFixture(overrides)
|
||||
createdSourceKeys.add(bundle.sourceKey)
|
||||
|
|
@ -267,7 +290,7 @@ describe('web speech acervo (C216)', () => {
|
|||
expect(byDuration.rows.map((row) => row.id)).toEqual([older])
|
||||
})
|
||||
|
||||
it('expands a theme only over the web rows and keeps the evidence (C192)', async () => {
|
||||
it('runs the sense engine only over the web rows and keeps the evidence (C229)', async () => {
|
||||
const runId = randomUUID().slice(0, 8)
|
||||
const marker = `temaweb${runId}`
|
||||
const id = await createWebSpeech({
|
||||
|
|
@ -279,25 +302,65 @@ describe('web speech acervo (C216)', () => {
|
|||
},
|
||||
],
|
||||
})
|
||||
await seedSpeechEmbedding(id, 'speech', null, [1, 0])
|
||||
await seedSpeechEmbedding(id, 'segment', 1, [1, 0])
|
||||
|
||||
const { coordinator } = await createUsers()
|
||||
const expansionCalls: string[] = []
|
||||
const embedCalls: string[] = []
|
||||
const data = await loadWebSpeechAcervoPageData(
|
||||
payload,
|
||||
coordinator,
|
||||
{ q: 'defesa do SUS', mode: 'tema' },
|
||||
async (theme) => {
|
||||
expansionCalls.push(theme)
|
||||
return { terms: [`atendimento universal ${marker}`] }
|
||||
{ source: 'internet', q: 'defesa do SUS', mode: 'tema' },
|
||||
async (query) => {
|
||||
embedCalls.push(query)
|
||||
return [1, 0]
|
||||
},
|
||||
)
|
||||
|
||||
expect(expansionCalls).toEqual(['defesa do SUS'])
|
||||
expect(embedCalls).toEqual(['defesa do SUS'])
|
||||
expect(data.themeUnavailable).toBe(false)
|
||||
expect(data.themeApplied).toBe(true)
|
||||
const row = data.rows.find((item) => item.id === id)
|
||||
expect(row).toBeDefined()
|
||||
expect(row?.excerpt.parts.some((part) => part.highlighted)).toBe(true)
|
||||
expect(row?.semanticMatch).toBe(true)
|
||||
expect(row?.matchedTextSearch).toBe(false)
|
||||
expect(row?.excerpt.parts.map((part) => part.text).join('')).toContain(marker)
|
||||
expect(row?.excerpt.parts.some((part) => part.highlighted)).toBe(false)
|
||||
})
|
||||
|
||||
it('lets an explicit order override relevance in the theme mode (C229)', async () => {
|
||||
const runId = randomUUID().slice(0, 8)
|
||||
const older = await createWebSpeech({
|
||||
speechAt: '2020-01-01T00:00',
|
||||
segments: [{ startSeconds: 0, endSeconds: 3, text: `Fala tema ${runId}` }],
|
||||
})
|
||||
const newer = await createWebSpeech({
|
||||
speechAt: '2026-01-01T00:00',
|
||||
segments: [{ startSeconds: 0, endSeconds: 3, text: `Fala tema ${runId}` }],
|
||||
})
|
||||
await seedSpeechEmbedding(older, 'speech', null, [1, 0])
|
||||
await seedSpeechEmbedding(newer, 'speech', null, [0.9, 0.435])
|
||||
|
||||
const { coordinator } = await createUsers()
|
||||
const byRelevance = await loadWebSpeechAcervoPageData(
|
||||
payload,
|
||||
coordinator,
|
||||
{ source: 'internet', q: `tema ${runId}`, mode: 'tema' },
|
||||
async () => [1, 0],
|
||||
)
|
||||
expect(byRelevance.rows.map((row) => row.id).indexOf(older)).toBeLessThan(
|
||||
byRelevance.rows.map((row) => row.id).indexOf(newer),
|
||||
)
|
||||
|
||||
const byDate = await loadWebSpeechAcervoPageData(
|
||||
payload,
|
||||
coordinator,
|
||||
{ source: 'internet', q: `tema ${runId}`, mode: 'tema', sort: 'recentes' },
|
||||
async () => [1, 0],
|
||||
)
|
||||
expect(byDate.state.sort).toBe('recentes')
|
||||
expect(byDate.rows.map((row) => row.id).indexOf(newer)).toBeLessThan(
|
||||
byDate.rows.map((row) => row.id).indexOf(older),
|
||||
)
|
||||
})
|
||||
|
||||
it('carries the private routes, the platform and the origin attribution in the detail', async () => {
|
||||
|
|
|
|||
|
|
@ -2,7 +2,13 @@
|
|||
|
||||
import { describe, expect, it } from 'vitest'
|
||||
|
||||
import { ACERVO_SORT_OPTIONS, acervoSortIsDuration, parseAcervoSort } from '@/lib/acervoListSort'
|
||||
import {
|
||||
ACERVO_SORT_OPTIONS,
|
||||
ACERVO_THEME_SORT_OPTIONS,
|
||||
acervoSortIsDuration,
|
||||
parseAcervoSort,
|
||||
parseAcervoThemeSort,
|
||||
} from '@/lib/acervoListSort'
|
||||
|
||||
// C216 — the shared ordering vocabulary of the acervo (recordings C219 + web
|
||||
// speeches): `recentes` is the default and never serialized, and only the two
|
||||
|
|
@ -37,3 +43,27 @@ describe('ACERVO_SORT_OPTIONS', () => {
|
|||
])
|
||||
})
|
||||
})
|
||||
|
||||
// C229 — the theme vocabulary of the speech acervo: relevance is the default
|
||||
// and `recentes` must remain askable; recordings/Central keep the list above.
|
||||
describe('theme sort vocabulary', () => {
|
||||
it('opens with relevance and keeps the shared options', () => {
|
||||
expect(ACERVO_THEME_SORT_OPTIONS).toEqual([
|
||||
{ value: 'relevancia', label: 'Mais relevantes' },
|
||||
...ACERVO_SORT_OPTIONS,
|
||||
])
|
||||
})
|
||||
|
||||
it('parses relevance and unknown values as the default', () => {
|
||||
expect(parseAcervoThemeSort('relevancia')).toBeUndefined()
|
||||
expect(parseAcervoThemeSort('maratona')).toBeUndefined()
|
||||
expect(parseAcervoThemeSort(undefined)).toBeUndefined()
|
||||
})
|
||||
|
||||
it('keeps recentes and the duration orders meaningful', () => {
|
||||
expect(parseAcervoThemeSort('recentes')).toBe('recentes')
|
||||
expect(parseAcervoThemeSort('duracao_maior')).toBe('duracao_maior')
|
||||
expect(parseAcervoThemeSort('duracao_menor')).toBe('duracao_menor')
|
||||
expect(parseAcervoThemeSort(['recentes', 'duracao_maior'])).toBe('recentes')
|
||||
})
|
||||
})
|
||||
|
|
|
|||
181
tests/unit/aiDeepInfraEmbed.unit.spec.ts
Normal file
181
tests/unit/aiDeepInfraEmbed.unit.spec.ts
Normal file
|
|
@ -0,0 +1,181 @@
|
|||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
|
||||
import {
|
||||
DEEPINFRA_EMBED_DIMENSIONS,
|
||||
DEEPINFRA_EMBED_MODEL,
|
||||
DEEPINFRA_EMBED_URL,
|
||||
EMBED_BATCH_SIZE,
|
||||
EMBED_TEXT_MAX_CHARS,
|
||||
embedSpeechQuery,
|
||||
embedSpeechTexts,
|
||||
} from '@/utilities/ai/deepInfraEmbed'
|
||||
|
||||
/**
|
||||
* C229 — the embeddings seam with `fetch` stubbed: the body carries the model
|
||||
* and the batched input, the key never rides the URL, every failure degrades to
|
||||
* `null` instead of throwing, and the vectors come back L2-normalized in input
|
||||
* order.
|
||||
*/
|
||||
|
||||
const fetchMock = vi.fn()
|
||||
|
||||
const vector = (head: number[]): number[] => [
|
||||
...head,
|
||||
...new Array(DEEPINFRA_EMBED_DIMENSIONS - head.length).fill(0),
|
||||
]
|
||||
|
||||
const jsonResponse = (payload: unknown, status = 200) =>
|
||||
({ ok: status >= 200 && status < 300, status, json: async () => payload }) as Response
|
||||
|
||||
const embeddingResponse = (vectors: number[][], promptTokens = 7) =>
|
||||
jsonResponse({
|
||||
data: vectors.map((embedding, index) => ({ index, embedding })),
|
||||
usage: { prompt_tokens: promptTokens },
|
||||
})
|
||||
|
||||
const stubFetch = (...responses: Response[]) => {
|
||||
fetchMock.mockReset()
|
||||
for (const response of responses) fetchMock.mockResolvedValueOnce(response)
|
||||
vi.stubGlobal('fetch', fetchMock)
|
||||
}
|
||||
|
||||
beforeEach(() => {
|
||||
process.env.DEEPINFRA_API_KEY = 'test-deepinfra-key'
|
||||
fetchMock.mockReset()
|
||||
vi.stubGlobal('fetch', fetchMock)
|
||||
})
|
||||
|
||||
afterEach(() => {
|
||||
delete process.env.DEEPINFRA_API_KEY
|
||||
vi.unstubAllGlobals()
|
||||
})
|
||||
|
||||
describe('embedSpeechTexts', () => {
|
||||
it('posts the model and the batched input with the bearer key', async () => {
|
||||
stubFetch(embeddingResponse([vector([3, 4])]))
|
||||
|
||||
const result = await embedSpeechTexts(['fala sobre o SUS'])
|
||||
|
||||
expect(result?.vectors).toEqual([vector([0.6, 0.8])])
|
||||
expect(result?.promptTokens).toBe(7)
|
||||
const [url, init] = fetchMock.mock.calls[0] as [string, { method: string; body: string }]
|
||||
expect(url).toBe(DEEPINFRA_EMBED_URL)
|
||||
expect(url).not.toContain('test-deepinfra-key')
|
||||
expect(init.method).toBe('POST')
|
||||
const body = JSON.parse(init.body) as Record<string, unknown>
|
||||
expect(body.model).toBe(DEEPINFRA_EMBED_MODEL)
|
||||
expect(body.input).toEqual(['fala sobre o SUS'])
|
||||
expect(body.encoding_format).toBe('float')
|
||||
})
|
||||
|
||||
it('returns an empty result without calling the provider', async () => {
|
||||
const result = await embedSpeechTexts([])
|
||||
|
||||
expect(result).toEqual({ vectors: [], promptTokens: 0 })
|
||||
expect(fetchMock).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('fails closed when the key is missing', async () => {
|
||||
delete process.env.DEEPINFRA_API_KEY
|
||||
|
||||
await expect(embedSpeechTexts(['fala'])).resolves.toBeNull()
|
||||
expect(fetchMock).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('keeps the input order when the provider answers out of order', async () => {
|
||||
stubFetch(
|
||||
jsonResponse({
|
||||
data: [
|
||||
{ index: 1, embedding: vector([0, 1]) },
|
||||
{ index: 0, embedding: vector([1, 0]) },
|
||||
],
|
||||
}),
|
||||
)
|
||||
|
||||
const result = await embedSpeechTexts(['primeira', 'segunda'])
|
||||
|
||||
expect(result?.vectors).toEqual([vector([1, 0]), vector([0, 1])])
|
||||
})
|
||||
|
||||
it('splits more than the batch size into sequential calls', async () => {
|
||||
const texts = new Array(EMBED_BATCH_SIZE + 1).fill('fala')
|
||||
stubFetch(
|
||||
embeddingResponse(texts.slice(0, EMBED_BATCH_SIZE).map(() => vector([1]))),
|
||||
embeddingResponse([vector([1])]),
|
||||
)
|
||||
|
||||
const result = await embedSpeechTexts(texts)
|
||||
|
||||
expect(result?.vectors).toHaveLength(EMBED_BATCH_SIZE + 1)
|
||||
expect(fetchMock).toHaveBeenCalledTimes(2)
|
||||
const second = JSON.parse((fetchMock.mock.calls[1][1] as { body: string }).body) as {
|
||||
input: string[]
|
||||
}
|
||||
expect(second.input).toHaveLength(1)
|
||||
})
|
||||
|
||||
it('truncates each text to the cost guard', async () => {
|
||||
stubFetch(embeddingResponse([vector([1])]))
|
||||
|
||||
await embedSpeechTexts(['x'.repeat(EMBED_TEXT_MAX_CHARS + 500)])
|
||||
|
||||
const body = JSON.parse((fetchMock.mock.calls[0][1] as { body: string }).body) as {
|
||||
input: string[]
|
||||
}
|
||||
expect(body.input[0]).toHaveLength(EMBED_TEXT_MAX_CHARS)
|
||||
})
|
||||
|
||||
it('rejects a wrong dimension, a non-finite value and a broken shape', async () => {
|
||||
stubFetch(jsonResponse({ data: [{ index: 0, embedding: [1, 0] }] }))
|
||||
await expect(embedSpeechTexts(['fala'])).resolves.toBeNull()
|
||||
|
||||
stubFetch(
|
||||
jsonResponse({
|
||||
data: [{ index: 0, embedding: vector([1]).map((v, i) => (i === 0 ? Number.NaN : v)) }],
|
||||
}),
|
||||
)
|
||||
await expect(embedSpeechTexts(['fala'])).resolves.toBeNull()
|
||||
|
||||
// Same length as the input, but the second answer repeats index 0: the
|
||||
// missing index must fail the batch (never a silent mix-up).
|
||||
stubFetch(
|
||||
jsonResponse({
|
||||
data: [
|
||||
{ index: 0, embedding: vector([1]) },
|
||||
{ index: 0, embedding: vector([1]) },
|
||||
],
|
||||
}),
|
||||
)
|
||||
await expect(embedSpeechTexts(['um', 'dois'])).resolves.toBeNull()
|
||||
})
|
||||
|
||||
it('maps an HTTP error and a thrown call to null', async () => {
|
||||
stubFetch(jsonResponse({ error: 'boom' }, 500))
|
||||
await expect(embedSpeechTexts(['fala'])).resolves.toBeNull()
|
||||
|
||||
fetchMock.mockReset()
|
||||
fetchMock.mockRejectedValue(new Error('network down'))
|
||||
vi.stubGlobal('fetch', fetchMock)
|
||||
await expect(embedSpeechTexts(['fala'])).resolves.toBeNull()
|
||||
})
|
||||
})
|
||||
|
||||
describe('embedSpeechQuery', () => {
|
||||
it('embeds a single trimmed query', async () => {
|
||||
stubFetch(embeddingResponse([vector([1])]))
|
||||
|
||||
await expect(embedSpeechQuery(' combate à oposição ')).resolves.toEqual(vector([1]))
|
||||
const body = JSON.parse((fetchMock.mock.calls[0][1] as { body: string }).body) as {
|
||||
input: string[]
|
||||
}
|
||||
expect(body.input).toEqual(['combate à oposição'])
|
||||
})
|
||||
|
||||
it('fails closed on a blank query and on a missing key', async () => {
|
||||
await expect(embedSpeechQuery(' ')).resolves.toBeNull()
|
||||
expect(fetchMock).not.toHaveBeenCalled()
|
||||
|
||||
delete process.env.DEEPINFRA_API_KEY
|
||||
await expect(embedSpeechQuery('impeachment')).resolves.toBeNull()
|
||||
})
|
||||
})
|
||||
134
tests/unit/speechIndex.unit.spec.ts
Normal file
134
tests/unit/speechIndex.unit.spec.ts
Normal file
|
|
@ -0,0 +1,134 @@
|
|||
// @vitest-environment node
|
||||
|
||||
import { describe, expect, it } from 'vitest'
|
||||
|
||||
import {
|
||||
formatSpeechIndexProbe,
|
||||
formatSpeechIndexReport,
|
||||
parseSpeechIndexArgs,
|
||||
speechIndexCostUsd,
|
||||
speechIndexEstimatedBytes,
|
||||
speechIndexReportStamp,
|
||||
} from '../../scripts/lib/speechIndex.mjs'
|
||||
|
||||
// C229 — the pure CLI contract of `pnpm acervo:index`: flags, the honest
|
||||
// report lines and the cost math. The DB/embedding path is covered by the int
|
||||
// suite and the probe by the runbook.
|
||||
|
||||
describe('parseSpeechIndexArgs', () => {
|
||||
it('defaults to the whole catalog with a dry report shape', () => {
|
||||
expect(parseSpeechIndexArgs([])).toEqual({
|
||||
source: 'all',
|
||||
limit: null,
|
||||
dryRun: false,
|
||||
force: false,
|
||||
probe: null,
|
||||
top: 10,
|
||||
out: 'data/acervo-index',
|
||||
help: false,
|
||||
})
|
||||
})
|
||||
|
||||
it('parses the write options', () => {
|
||||
expect(
|
||||
parseSpeechIndexArgs(['--source', 'web', '--limit', '20', '--dry-run', '--force']),
|
||||
).toMatchObject({ source: 'web', limit: 20, dryRun: true, force: true })
|
||||
})
|
||||
|
||||
it('parses the probe with its own top', () => {
|
||||
expect(parseSpeechIndexArgs(['--probe', 'combate à oposição', '--top', '5'])).toMatchObject({
|
||||
probe: 'combate à oposição',
|
||||
top: 5,
|
||||
})
|
||||
})
|
||||
|
||||
it('refuses unknown sources, invalid numbers and escaping out dirs', () => {
|
||||
expect(() => parseSpeechIndexArgs(['--source', 'gravacoes'])).toThrow(/--source/)
|
||||
expect(() => parseSpeechIndexArgs(['--limit', '0'])).toThrow(/--limit/)
|
||||
expect(() => parseSpeechIndexArgs(['--top', 'x', '--probe', 'tema'])).toThrow(/--top/)
|
||||
expect(() => parseSpeechIndexArgs(['--out', '../fora'])).toThrow(/--out/)
|
||||
})
|
||||
|
||||
it('refuses mixing the read-only probe with write options', () => {
|
||||
expect(() => parseSpeechIndexArgs(['--probe', 'tema', '--force'])).toThrow(/--probe/)
|
||||
expect(() => parseSpeechIndexArgs(['--probe', 'tema', '--limit', '3'])).toThrow(/--probe/)
|
||||
expect(() => parseSpeechIndexArgs(['--top', '3'])).toThrow(/--top só vale com --probe/)
|
||||
})
|
||||
})
|
||||
|
||||
describe('speechIndexCostUsd', () => {
|
||||
it('multiplies the tokens by the model price per million', () => {
|
||||
expect(speechIndexCostUsd(1_000_000, 0.01)).toBeCloseTo(0.01)
|
||||
expect(speechIndexCostUsd(250_000, 0.01)).toBeCloseTo(0.0025)
|
||||
expect(speechIndexCostUsd(0, 0.01)).toBe(0)
|
||||
})
|
||||
})
|
||||
|
||||
describe('speechIndexEstimatedBytes', () => {
|
||||
it('estimates the JSON cost of the vectors', () => {
|
||||
expect(speechIndexEstimatedBytes(0, 1024)).toBe(0)
|
||||
expect(speechIndexEstimatedBytes(1000, 1024)).toBe(8_192_000)
|
||||
})
|
||||
})
|
||||
|
||||
describe('formatSpeechIndexReport', () => {
|
||||
const base = {
|
||||
target: 'localhost/teqo_wt229',
|
||||
source: 'all',
|
||||
speeches: 100,
|
||||
upToDate: 80,
|
||||
skippedNoText: 5,
|
||||
toIndex: 15,
|
||||
units: 300,
|
||||
durationMs: 12_340,
|
||||
}
|
||||
|
||||
it('renders the dry-run preview without write claims', () => {
|
||||
const lines = formatSpeechIndexReport({
|
||||
...base,
|
||||
mode: 'dry-run',
|
||||
estimatedBytes: 8_192_000,
|
||||
})
|
||||
|
||||
expect(lines[0]).toContain('modo: dry-run | fonte: all')
|
||||
expect(lines[1]).toContain('falas: 100 | atualizadas: 80 | sem texto: 5 | a indexar: 15')
|
||||
expect(lines[2]).toContain('trechos a vetorizar: 300')
|
||||
expect(lines[2]).toContain('bytes estimados: 8.2 MB')
|
||||
expect(lines[3]).toContain('duração: 12.3s')
|
||||
expect(lines.join('\n')).not.toContain('custo')
|
||||
})
|
||||
|
||||
it('renders the write report with indexed, failures, tokens and cost', () => {
|
||||
const lines = formatSpeechIndexReport({
|
||||
...base,
|
||||
mode: 'index',
|
||||
indexed: 15,
|
||||
failed: 0,
|
||||
promptTokens: 123_456,
|
||||
costUsd: 0.0012,
|
||||
})
|
||||
|
||||
expect(lines[2]).toContain('indexadas: 15 | falhas: 0 | trechos vetorizados: 300')
|
||||
expect(lines[2]).toContain('tokens: 123456')
|
||||
expect(lines[2]).toContain('custo estimado: US$ 0.0012')
|
||||
})
|
||||
})
|
||||
|
||||
describe('formatSpeechIndexProbe', () => {
|
||||
it('renders the ranked calibration lines', () => {
|
||||
const lines = formatSpeechIndexProbe('impeachment', [
|
||||
{ score: 0.71234, speechAt: '2016-04-17T20:00', origin: 'camara', sourceKey: 'camara:1' },
|
||||
{ score: 0.5011, speechAt: '2024-01-01T10:00', origin: 'web', sourceKey: 'web:youtube:x' },
|
||||
])
|
||||
|
||||
expect(lines[0]).toBe('[acervo:index] probe "impeachment" — top 2')
|
||||
expect(lines[1]).toBe('1. 0.7123 · 2016-04-17T20:00 · camara · camara:1')
|
||||
expect(lines[2]).toBe('2. 0.5011 · 2024-01-01T10:00 · web · web:youtube:x')
|
||||
})
|
||||
})
|
||||
|
||||
describe('speechIndexReportStamp', () => {
|
||||
it('turns the ISO instant into a filesystem-safe stamp', () => {
|
||||
expect(speechIndexReportStamp('2026-09-26T03:41:18.123Z')).toBe('2026-09-26T03-41-18-123Z')
|
||||
})
|
||||
})
|
||||
|
|
@ -3,6 +3,7 @@
|
|||
import { describe, expect, it } from 'vitest'
|
||||
|
||||
import {
|
||||
buildSpeechFacetWhere,
|
||||
buildSpeechListWhere,
|
||||
buildSpeechListWhereIncludingCutOrigins,
|
||||
} from '@/utilities/speech/speechListFilters'
|
||||
|
|
@ -132,63 +133,34 @@ describe('buildSpeechListWhereIncludingCutOrigins (C174)', () => {
|
|||
})
|
||||
})
|
||||
|
||||
describe('buildSpeechListWhere theme expansion (C192)', () => {
|
||||
it('ORs the query with every expanded theme term', () => {
|
||||
const where = buildSpeechListWhere(parseSpeechListParams({ q: 'defesa do SUS' }), [
|
||||
'saúde pública',
|
||||
'SUS',
|
||||
])
|
||||
|
||||
expect(where).toEqual({
|
||||
and: [
|
||||
camara,
|
||||
{
|
||||
or: [
|
||||
{ searchText: { like: 'defesa do sus' } },
|
||||
{ keywords: { contains: 'defesa do SUS' } },
|
||||
{ searchText: { like: 'saude publica' } },
|
||||
{ keywords: { contains: 'saúde pública' } },
|
||||
{ searchText: { like: 'sus' } },
|
||||
{ keywords: { contains: 'SUS' } },
|
||||
],
|
||||
},
|
||||
],
|
||||
})
|
||||
})
|
||||
|
||||
it('dedupes a term that repeats the query', () => {
|
||||
expect(buildSpeechListWhere(parseSpeechListParams({ q: 'SUS' }), ['SUS'])).toEqual({
|
||||
and: [camara, { or: [{ searchText: { like: 'sus' } }, { keywords: { contains: 'SUS' } }] }],
|
||||
})
|
||||
})
|
||||
|
||||
it('keeps the facets AND-ed and the cut origins OR-ed alongside the expansion', () => {
|
||||
// C229 — the semantic theme path seeds its candidates from this facet-only
|
||||
// boundary (no textual branch); the literal where above builds on it.
|
||||
describe('buildSpeechFacetWhere (C229)', () => {
|
||||
it('returns the source discriminator and every facet branch', () => {
|
||||
expect(
|
||||
buildSpeechListWhereIncludingCutOrigins(
|
||||
parseSpeechListParams({ q: 'reforma', topic: ['saude'] }),
|
||||
[7],
|
||||
['saúde'],
|
||||
buildSpeechFacetWhere(
|
||||
parseSpeechListParams({
|
||||
year: '2026',
|
||||
topic: ['saude'],
|
||||
scope: ['bahia'],
|
||||
phase: ['Breves Comunicações'],
|
||||
municipality: '12',
|
||||
duration: ['curta'],
|
||||
}),
|
||||
),
|
||||
).toEqual({
|
||||
and: [
|
||||
camara,
|
||||
{ topics: { in: ['saude'] } },
|
||||
{
|
||||
or: [
|
||||
{ searchText: { like: 'reforma' } },
|
||||
{ keywords: { contains: 'reforma' } },
|
||||
{ searchText: { like: 'saude' } },
|
||||
{ keywords: { contains: 'saúde' } },
|
||||
{ id: { in: [7] } },
|
||||
],
|
||||
},
|
||||
],
|
||||
})
|
||||
).toEqual([
|
||||
camara,
|
||||
{ year: { in: [2026] } },
|
||||
{ topics: { in: ['saude'] } },
|
||||
{ scopes: { in: ['bahia'] } },
|
||||
{ phase: { in: ['Breves Comunicações'] } },
|
||||
{ mentionedMunicipalities: { in: [12] } },
|
||||
{ durationSeconds: { less_than: 120 } },
|
||||
])
|
||||
})
|
||||
|
||||
it('stays byte-identical to today without theme terms', () => {
|
||||
const state = parseSpeechListParams({ q: 'reforma' })
|
||||
expect(buildSpeechListWhere(state, [])).toEqual(buildSpeechListWhere(state))
|
||||
it('never carries the textual query branch', () => {
|
||||
expect(buildSpeechFacetWhere(parseSpeechListParams({ q: 'SUS' }))).toEqual([camara])
|
||||
})
|
||||
})
|
||||
|
||||
|
|
|
|||
|
|
@ -238,3 +238,62 @@ describe('web speeches source (C216)', () => {
|
|||
expect(speechHasActiveFilters(parseSpeechListParams({}))).toBe(false)
|
||||
})
|
||||
})
|
||||
|
||||
// C229 — the theme mode of the web source: relevance is the default and
|
||||
// `recentes` becomes a real state that must serialize; the exact mode keeps
|
||||
// its bytes and the Câmara keeps no sort at all.
|
||||
describe('theme sort of the web source (C229)', () => {
|
||||
const themeParams = { source: 'internet', q: 'combate à oposição', mode: 'tema' } as const
|
||||
|
||||
it('parses the theme sort with relevance and unknown values as the default', () => {
|
||||
expect(parseSpeechListParams({ ...themeParams, sort: 'recentes' }).sort).toBe('recentes')
|
||||
expect(parseSpeechListParams({ ...themeParams, sort: 'duracao_maior' }).sort).toBe(
|
||||
'duracao_maior',
|
||||
)
|
||||
expect(parseSpeechListParams({ ...themeParams, sort: 'relevancia' }).sort).toBeUndefined()
|
||||
expect(parseSpeechListParams({ ...themeParams, sort: 'maratona' }).sort).toBeUndefined()
|
||||
// The Câmara contract never carries a sort, not even in the theme mode.
|
||||
expect(parseSpeechListParams({ q: 'SUS', mode: 'tema', sort: 'recentes' }).sort).toBeUndefined()
|
||||
})
|
||||
|
||||
it('serializes an explicit recentes in the theme mode only', () => {
|
||||
expect(
|
||||
serializeCanonicalSpeechListSearchParams(
|
||||
parseSpeechListParams({ ...themeParams, sort: 'recentes' }),
|
||||
).toString(),
|
||||
).toBe('source=internet&q=combate+%C3%A0+oposi%C3%A7%C3%A3o&mode=tema&sort=recentes')
|
||||
// Relevance and unknown values canonicalize away.
|
||||
expect(
|
||||
serializeCanonicalSpeechListSearchParams(
|
||||
parseSpeechListParams({ ...themeParams, sort: 'relevancia' }),
|
||||
).get('sort'),
|
||||
).toBeNull()
|
||||
// The exact mode still omits its own default.
|
||||
expect(
|
||||
serializeCanonicalSpeechListSearchParams(
|
||||
parseSpeechListParams({ source: 'internet', q: 'SUS', sort: 'recentes' }),
|
||||
).get('sort'),
|
||||
).toBeNull()
|
||||
})
|
||||
|
||||
it('canonicalizes relevance and unknown theme sorts into a redirect', () => {
|
||||
const relevance = resolveSpeechListUrl({ ...themeParams, sort: 'relevancia' })
|
||||
expect(relevance.state.sort).toBeUndefined()
|
||||
expect(relevance.redirectHref).toBe(
|
||||
'/campanha/comunicacao/acervo?source=internet&q=combate+%C3%A0+oposi%C3%A7%C3%A3o&mode=tema',
|
||||
)
|
||||
|
||||
const unknown = resolveSpeechListUrl({ ...themeParams, sort: 'maratona' })
|
||||
expect(unknown.redirectHref).toBe(
|
||||
'/campanha/comunicacao/acervo?source=internet&q=combate+%C3%A0+oposi%C3%A7%C3%A3o&mode=tema',
|
||||
)
|
||||
})
|
||||
|
||||
it('keeps the canonical round-trip of an explicit theme order', () => {
|
||||
const state = parseSpeechListParams({ ...themeParams, sort: 'recentes' })
|
||||
expect(buildSpeechListHref(state, 2)).toBe(
|
||||
'/campanha/comunicacao/acervo?source=internet&q=combate+%C3%A0+oposi%C3%A7%C3%A3o&mode=tema&sort=recentes&page=2',
|
||||
)
|
||||
expect(resolveSpeechListUrl({ ...themeParams, sort: 'recentes' }).redirectHref).toBeUndefined()
|
||||
})
|
||||
})
|
||||
|
|
|
|||
|
|
@ -62,7 +62,8 @@ const speech = (overrides: Partial<SpeechListItemViewModel> = {}): SpeechListIte
|
|||
officialTextUrl: null,
|
||||
cuts: [],
|
||||
matchedTextSearch: true,
|
||||
themeMatchTerm: null,
|
||||
semanticMatch: false,
|
||||
literalFallback: false,
|
||||
...overrides,
|
||||
})
|
||||
|
||||
|
|
@ -88,6 +89,9 @@ const webSpeech = (
|
|||
],
|
||||
thumbnailUrl: null,
|
||||
watchHref: '/campanha/comunicacao/acervo/internet/9',
|
||||
semanticMatch: false,
|
||||
literalFallback: false,
|
||||
matchedTextSearch: false,
|
||||
...overrides,
|
||||
})
|
||||
|
||||
|
|
@ -115,18 +119,70 @@ describe('speech result cards excerpt', () => {
|
|||
expect(html).not.toContain('text-sm leading-relaxed')
|
||||
})
|
||||
|
||||
it('wraps the theme provenance excerpt in quotes', () => {
|
||||
it('shows the semantic evidence block with the seal and without a highlight', () => {
|
||||
const html = renderWithAppRouter(
|
||||
createElement(SpeechResultCard, { speech: speech({ themeMatchTerm: 'saúde' }) }),
|
||||
createElement(SpeechResultCard, {
|
||||
speech: speech({
|
||||
semanticMatch: true,
|
||||
excerpt: {
|
||||
parts: [{ text: 'O embate com a oposição', highlighted: false }],
|
||||
truncatedStart: false,
|
||||
truncatedEnd: true,
|
||||
},
|
||||
}),
|
||||
}),
|
||||
)
|
||||
|
||||
expect(html).toContain('Por que apareceu')
|
||||
expect(html).toContain('Trecho mais próximo do tema')
|
||||
expect(html).toContain('Tema')
|
||||
expect(html).toContain('“')
|
||||
expect(html).toContain('”')
|
||||
// The provenance replaces the common excerpt — they never coexist (C192).
|
||||
expect(html).not.toContain('<mark')
|
||||
// The evidence block replaces the common excerpt — they never coexist.
|
||||
expect(html).not.toContain('text-sm leading-relaxed')
|
||||
})
|
||||
|
||||
it('shows the term seal alongside the theme seal when the query also matches', () => {
|
||||
const html = renderWithAppRouter(
|
||||
createElement(SpeechResultCard, {
|
||||
speech: speech({ semanticMatch: true, matchedTextSearch: true }),
|
||||
}),
|
||||
)
|
||||
|
||||
expect(html).toContain('Tema')
|
||||
expect(html).toContain('Termo exato')
|
||||
})
|
||||
|
||||
it('shows the same semantic evidence on the web card', () => {
|
||||
const html = renderWithAppRouter(
|
||||
createElement(WebSpeechResultCard, {
|
||||
speech: webSpeech({
|
||||
semanticMatch: true,
|
||||
excerpt: {
|
||||
parts: [{ text: 'O embate com a oposição', highlighted: false }],
|
||||
truncatedStart: false,
|
||||
truncatedEnd: true,
|
||||
},
|
||||
}),
|
||||
}),
|
||||
)
|
||||
|
||||
expect(html).toContain('Trecho mais próximo do tema')
|
||||
expect(html).toContain('Tema')
|
||||
expect(html).not.toContain('<mark')
|
||||
expect(html).toContain('Fala na internet')
|
||||
})
|
||||
|
||||
it('labels a degraded literal card with the term seal', () => {
|
||||
const html = renderWithAppRouter(
|
||||
createElement(SpeechResultCard, { speech: speech({ literalFallback: true }) }),
|
||||
)
|
||||
|
||||
expect(html).toContain('data-testid="speech-literal-badge"')
|
||||
expect(html).toContain('Termo exato')
|
||||
expect(html).not.toContain('Trecho mais próximo do tema')
|
||||
})
|
||||
|
||||
it('keeps the web card own excerpt classes', () => {
|
||||
const html = renderWithAppRouter(createElement(WebSpeechResultCard, { speech: webSpeech() }))
|
||||
|
||||
|
|
|
|||
249
tests/unit/speechSemantic.unit.spec.ts
Normal file
249
tests/unit/speechSemantic.unit.spec.ts
Normal file
|
|
@ -0,0 +1,249 @@
|
|||
// @vitest-environment node
|
||||
|
||||
import { describe, expect, it } from 'vitest'
|
||||
|
||||
import {
|
||||
cosineSimilarity,
|
||||
dotProduct,
|
||||
meanPoolVectors,
|
||||
normalizeVector,
|
||||
pickBestSemanticChunk,
|
||||
planSpeechIndex,
|
||||
rankSemanticHits,
|
||||
sortSemanticHits,
|
||||
SPEECH_SEMANTIC_MIN_COSINE,
|
||||
speechEmbeddingHash,
|
||||
speechSemanticSourceText,
|
||||
speechTextWindows,
|
||||
type SpeechSemanticCandidate,
|
||||
type SpeechSemanticHit,
|
||||
} from '@/lib/speechSemantic'
|
||||
|
||||
// C229 — the pure engine of the semantic acervo: vector math, text windows,
|
||||
// the content hash and the ranking rules (threshold + relevance default with
|
||||
// the explicit order overriding it). No network and no Payload here.
|
||||
|
||||
describe('vector math', () => {
|
||||
it('computes the dot product and rejects mismatched dimensions', () => {
|
||||
expect(dotProduct([1, 2, 3], [4, 5, 6])).toBe(32)
|
||||
expect(dotProduct([1, 2], [1])).toBe(0)
|
||||
})
|
||||
|
||||
it('normalizes to unit length and keeps a zero vector finite', () => {
|
||||
expect(normalizeVector([3, 4])).toEqual([0.6, 0.8])
|
||||
expect(normalizeVector([0, 0])).toEqual([0, 0])
|
||||
})
|
||||
|
||||
it('computes a real cosine regardless of scale', () => {
|
||||
expect(cosineSimilarity([1, 0], [1, 0])).toBeCloseTo(1)
|
||||
expect(cosineSimilarity([1, 0], [0, 1])).toBeCloseTo(0)
|
||||
expect(cosineSimilarity([1, 0], [-1, 0])).toBeCloseTo(-1)
|
||||
expect(cosineSimilarity([2, 0], [7, 0])).toBeCloseTo(1)
|
||||
expect(cosineSimilarity([1, 2], [1])).toBe(0)
|
||||
expect(cosineSimilarity([0, 0], [1, 1])).toBe(0)
|
||||
})
|
||||
|
||||
it('pools vectors by the normalized mean and skips unusable shapes', () => {
|
||||
expect(
|
||||
meanPoolVectors([
|
||||
[1, 0],
|
||||
[1, 0],
|
||||
]),
|
||||
).toEqual([1, 0])
|
||||
// The mean of opposite directions is the zero vector, never NaN.
|
||||
expect(
|
||||
meanPoolVectors([
|
||||
[1, 0],
|
||||
[-1, 0],
|
||||
]),
|
||||
).toEqual([0, 0])
|
||||
expect(
|
||||
meanPoolVectors([
|
||||
[1, 0],
|
||||
[0, 1, 0],
|
||||
]),
|
||||
).toEqual([1, 0])
|
||||
expect(meanPoolVectors([])).toEqual([])
|
||||
})
|
||||
})
|
||||
|
||||
describe('speechTextWindows', () => {
|
||||
const longText = 'palavra '.repeat(400).trim()
|
||||
|
||||
it('keeps a short text as a single window', () => {
|
||||
expect(speechTextWindows('fala curta')).toEqual([
|
||||
{ order: 0, start: 0, end: 10, text: 'fala curta' },
|
||||
])
|
||||
expect(speechTextWindows('')).toEqual([])
|
||||
})
|
||||
|
||||
it('splits long text into ordered overlapping windows without cutting words', () => {
|
||||
const windows = speechTextWindows(longText, { size: 100, overlap: 20 })
|
||||
|
||||
expect(windows.length).toBeGreaterThan(2)
|
||||
expect(windows.map((window) => window.order)).toEqual(windows.map((_, index) => index))
|
||||
for (const window of windows) {
|
||||
expect(window.text.length).toBeLessThanOrEqual(100)
|
||||
expect(window.text).toBe(window.text.trim())
|
||||
// No window starts or ends mid-word: the slice boundaries are word edges.
|
||||
expect(longText.slice(window.start, window.end)).toBe(
|
||||
`${longText.slice(window.start, window.end).trim()}`,
|
||||
)
|
||||
}
|
||||
// Overlap: the next window starts before the previous one ended.
|
||||
expect(windows[1].start).toBeLessThan(windows[0].end)
|
||||
// Full coverage: every character of the text belongs to some window.
|
||||
let covered = 0
|
||||
for (const window of windows) covered = Math.max(covered, window.end)
|
||||
expect(covered).toBe(longText.length)
|
||||
})
|
||||
})
|
||||
|
||||
describe('speechEmbeddingHash', () => {
|
||||
it('is deterministic, text-sensitive and model-sensitive', () => {
|
||||
expect(speechEmbeddingHash('fala', 'bge')).toBe(speechEmbeddingHash('fala', 'bge'))
|
||||
expect(speechEmbeddingHash('fala', 'bge')).not.toBe(speechEmbeddingHash('outra', 'bge'))
|
||||
expect(speechEmbeddingHash('fala', 'bge')).not.toBe(speechEmbeddingHash('fala', 'outro'))
|
||||
})
|
||||
})
|
||||
|
||||
describe('rankSemanticHits', () => {
|
||||
const candidates: SpeechSemanticCandidate[] = [
|
||||
{ id: 1, speechAt: '2026-01-01T10:00', vector: [1, 0] }, // 1.0
|
||||
{ id: 2, speechAt: '2026-02-01T10:00', vector: [0.8, 0.6] }, // 0.8
|
||||
{ id: 3, speechAt: '2026-03-01T10:00', vector: [0.8, 0.6] }, // 0.8, newer wins the tie
|
||||
{ id: 4, speechAt: '2026-04-01T10:00', vector: [0, 1] }, // 0.0 (below the cut-off)
|
||||
{ id: 5, speechAt: '2026-05-01T10:00', vector: null },
|
||||
{ id: 6, speechAt: '2026-06-01T10:00', vector: [1, 0, 0] },
|
||||
{ id: 7, speechAt: '2026-07-01T10:00', vector: [0.4, 0.916515] }, // ~0.4 (boundary, kept)
|
||||
]
|
||||
const query = [1, 0]
|
||||
|
||||
it('drops unusable candidates and the scores below the cut-off', () => {
|
||||
const hits = rankSemanticHits(query, candidates)
|
||||
expect(hits.map((hit) => hit.id)).toEqual([1, 3, 2, 7])
|
||||
})
|
||||
|
||||
it('sorts by score desc with the date as tiebreak and keeps the duration', () => {
|
||||
const hits = rankSemanticHits(query, [
|
||||
{ id: 9, speechAt: '2026-01-01T10:00', durationSeconds: 120, vector: [1, 0] },
|
||||
{ id: 8, speechAt: '2026-01-01T10:00', durationSeconds: null, vector: [1, 0] },
|
||||
])
|
||||
expect(hits.map((hit) => hit.id)).toEqual([8, 9])
|
||||
expect(hits[0]).toMatchObject({ durationSeconds: null, score: 1 })
|
||||
expect(hits[1]).toMatchObject({ durationSeconds: 120 })
|
||||
})
|
||||
|
||||
it('honors a custom cut-off', () => {
|
||||
expect(rankSemanticHits(query, candidates, { minScore: 0.9 }).map((hit) => hit.id)).toEqual([1])
|
||||
expect(
|
||||
rankSemanticHits([1, 0], [{ id: 1, speechAt: 'x', vector: [1, 0] }], { minScore: 0.95 }),
|
||||
).toHaveLength(1)
|
||||
})
|
||||
|
||||
it('keeps the documented default cut-off', () => {
|
||||
expect(SPEECH_SEMANTIC_MIN_COSINE).toBe(0.4)
|
||||
})
|
||||
})
|
||||
|
||||
describe('sortSemanticHits', () => {
|
||||
const hits: SpeechSemanticHit[] = [
|
||||
{ id: 1, speechAt: '2026-01-01T10:00', durationSeconds: 600, score: 0.9 },
|
||||
{ id: 3, speechAt: '2026-02-01T10:00', durationSeconds: null, score: 0.7 },
|
||||
{ id: 2, speechAt: '2026-03-01T10:00', durationSeconds: 120, score: 0.5 },
|
||||
]
|
||||
|
||||
it('keeps the relevance order by default', () => {
|
||||
expect(sortSemanticHits(hits, 'relevancia').map((hit) => hit.id)).toEqual([1, 3, 2])
|
||||
})
|
||||
|
||||
it('orders by date when the user asks for recentes', () => {
|
||||
expect(sortSemanticHits(hits, 'recentes').map((hit) => hit.id)).toEqual([2, 3, 1])
|
||||
})
|
||||
|
||||
it('orders by duration in both directions', () => {
|
||||
expect(sortSemanticHits(hits, 'duracao_maior').map((hit) => hit.id)).toEqual([1, 2, 3])
|
||||
expect(sortSemanticHits(hits, 'duracao_menor').map((hit) => hit.id)).toEqual([2, 1, 3])
|
||||
})
|
||||
})
|
||||
|
||||
describe('pickBestSemanticChunk', () => {
|
||||
it('picks the closest usable chunk and breaks ties by the lowest key', () => {
|
||||
expect(
|
||||
pickBestSemanticChunk(
|
||||
[1, 0],
|
||||
[
|
||||
{ key: 20, vector: [0, 1] },
|
||||
{ key: 10, vector: [1, 0] },
|
||||
{ key: 30, vector: [1, 0] },
|
||||
],
|
||||
),
|
||||
).toEqual({ key: 10, score: 1 })
|
||||
})
|
||||
|
||||
it('returns null when no chunk can be compared', () => {
|
||||
expect(pickBestSemanticChunk([1, 0], [])).toBeNull()
|
||||
expect(pickBestSemanticChunk([1, 0], [{ key: 1, vector: null }])).toBeNull()
|
||||
expect(pickBestSemanticChunk([1, 0], [{ key: 1, vector: [1, 0, 0] }])).toBeNull()
|
||||
})
|
||||
})
|
||||
|
||||
describe('speechSemanticSourceText', () => {
|
||||
it('prefers the official transcript, then the summary, then the search text', () => {
|
||||
expect(
|
||||
speechSemanticSourceText({
|
||||
officialTranscript: 'Transcrição',
|
||||
summary: 'Resumo',
|
||||
searchText: 'busca',
|
||||
}),
|
||||
).toBe('Transcrição')
|
||||
expect(speechSemanticSourceText({ officialTranscript: ' ', summary: 'Resumo' })).toBe('Resumo')
|
||||
expect(speechSemanticSourceText({ searchText: 'busca' })).toBe('busca')
|
||||
expect(speechSemanticSourceText({})).toBeNull()
|
||||
})
|
||||
})
|
||||
|
||||
describe('planSpeechIndex', () => {
|
||||
const model = 'test-model'
|
||||
|
||||
it('indexes the ASR segments by their own order and skips blanks', () => {
|
||||
const plan = planSpeechIndex(
|
||||
{
|
||||
segments: [
|
||||
{ order: 1, startSeconds: 0, text: 'Primeiro trecho' },
|
||||
{ order: 2, startSeconds: 3, text: ' ' },
|
||||
{ order: 3, startSeconds: 6, text: 'Terceiro trecho' },
|
||||
],
|
||||
},
|
||||
{ model },
|
||||
)
|
||||
|
||||
expect(plan?.units).toEqual([
|
||||
{ kind: 'segment', order: 1, startSeconds: 0, text: 'Primeiro trecho' },
|
||||
{ kind: 'segment', order: 3, startSeconds: 6, text: 'Terceiro trecho' },
|
||||
])
|
||||
expect(plan?.speechHash).toBe(
|
||||
speechEmbeddingHash('segment:1:Primeiro trecho\nsegment:3:Terceiro trecho', model),
|
||||
)
|
||||
})
|
||||
|
||||
it('falls back to text windows when the speech has no segments', () => {
|
||||
const plan = planSpeechIndex({ officialTranscript: 'fala '.repeat(400) }, { model })
|
||||
|
||||
expect(plan?.units.length).toBeGreaterThan(1)
|
||||
expect(plan?.units.every((unit) => unit.kind === 'window')).toBe(true)
|
||||
expect(plan?.units[0]).toMatchObject({ order: 0, startSeconds: null })
|
||||
})
|
||||
|
||||
it('returns null without any usable text', () => {
|
||||
expect(planSpeechIndex({ segments: [] }, { model })).toBeNull()
|
||||
expect(planSpeechIndex({ officialTranscript: ' ', summary: null }, { model })).toBeNull()
|
||||
})
|
||||
|
||||
it('changes the hash with the model so a model swap reindexes', () => {
|
||||
const speech = { segments: [{ order: 1, startSeconds: 0, text: 'Fala' }] }
|
||||
expect(planSpeechIndex(speech, { model: 'a' })?.speechHash).not.toBe(
|
||||
planSpeechIndex(speech, { model: 'b' })?.speechHash,
|
||||
)
|
||||
})
|
||||
})
|
||||
|
|
@ -69,18 +69,22 @@ describe('toSpeechListItemViewModel thumbnail (C182 — frame of the speech)', (
|
|||
})
|
||||
})
|
||||
|
||||
describe('toSpeechListItemViewModel theme match (C192)', () => {
|
||||
const segment = (text: string) => ({ startSeconds: 0, endSeconds: 3, text })
|
||||
const themeRow = ({
|
||||
describe('toSpeechListItemViewModel semantic evidence (C229)', () => {
|
||||
const segment = (text: string, startSeconds = 0) => ({
|
||||
startSeconds,
|
||||
endSeconds: startSeconds + 3,
|
||||
text,
|
||||
})
|
||||
const semanticRow = ({
|
||||
text,
|
||||
query,
|
||||
themeTerms,
|
||||
evidence,
|
||||
keywords = [],
|
||||
searchText,
|
||||
}: {
|
||||
text: string
|
||||
query?: string
|
||||
themeTerms: string[]
|
||||
evidence?: { text: string; startSeconds: number | null } | null
|
||||
keywords?: string[]
|
||||
searchText?: string
|
||||
}) =>
|
||||
|
|
@ -89,94 +93,75 @@ describe('toSpeechListItemViewModel theme match (C192)', () => {
|
|||
segments: [segment(text)],
|
||||
query,
|
||||
municipalityLabels: new Map(),
|
||||
themeTerms,
|
||||
semanticMatch: Boolean(evidence),
|
||||
semanticEvidence: evidence,
|
||||
})
|
||||
|
||||
it('surfaces the theme passage with the expanded term highlighted', () => {
|
||||
const row = themeRow({
|
||||
text: 'Defendemos o atendimento público e acesso universal à saúde',
|
||||
searchText: 'defendemos o atendimento publico e acesso universal a saude',
|
||||
query: 'defesa do SUS',
|
||||
themeTerms: ['acesso universal'],
|
||||
it('shows the semantic passage with no highlight when no term matches', () => {
|
||||
const row = semanticRow({
|
||||
text: 'Uma fala sobre outro assunto',
|
||||
query: 'combate à oposição',
|
||||
evidence: { text: 'O embate com a oposição e o bolsonarismo', startSeconds: 42 },
|
||||
})
|
||||
|
||||
expect(row.matchKind).toBe('theme')
|
||||
expect(row.themeMatchTerm).toBe('acesso universal')
|
||||
expect(row.semanticMatch).toBe(true)
|
||||
expect(row.matchedTextSearch).toBe(false)
|
||||
expect(row.excerpt.parts.some((part) => part.highlighted)).toBe(true)
|
||||
expect(row.excerpt.parts.some((part) => part.highlighted)).toBe(false)
|
||||
expect(row.excerpt.parts.map((part) => part.text).join('')).toContain('bolsonarismo')
|
||||
expect(row.watchHref).toBe(
|
||||
'/campanha/comunicacao/acervo/1?t=42&q=combate+%C3%A0+oposi%C3%A7%C3%A3o',
|
||||
)
|
||||
})
|
||||
|
||||
it('keeps the theme term when the speech also matches the literal query', () => {
|
||||
const row = themeRow({
|
||||
it('keeps the literal match flag when the speech also contains the query', () => {
|
||||
const row = semanticRow({
|
||||
text: 'A defesa do SUS e da saúde pública baiana',
|
||||
searchText: 'a defesa do sus e da saude publica baiana',
|
||||
query: 'SUS',
|
||||
themeTerms: ['saúde pública'],
|
||||
evidence: { text: 'A defesa do SUS', startSeconds: 0 },
|
||||
})
|
||||
|
||||
expect(row.matchKind).toBe('segment')
|
||||
expect(row.themeMatchTerm).toBe('saúde pública')
|
||||
expect(row.semanticMatch).toBe(true)
|
||||
expect(row.matchedTextSearch).toBe(true)
|
||||
// The evidence never highlights the term: the engine did not match it.
|
||||
expect(row.excerpt.parts.some((part) => part.highlighted)).toBe(false)
|
||||
})
|
||||
|
||||
it('matches a keyword as theme evidence and uses it as the excerpt', () => {
|
||||
const row = themeRow({
|
||||
text: 'Uma fala sobre outro assunto',
|
||||
query: 'defesa do SUS',
|
||||
themeTerms: ['Farmácia Popular'],
|
||||
keywords: ['Farmácia Popular'],
|
||||
it('stays literal without semantic evidence', () => {
|
||||
const row = semanticRow({
|
||||
text: 'A defesa do SUS',
|
||||
searchText: 'a defesa do sus',
|
||||
query: 'SUS',
|
||||
})
|
||||
|
||||
expect(row.matchKind).toBe('theme')
|
||||
expect(row.themeMatchTerm).toBe('Farmácia Popular')
|
||||
const excerptText = row.excerpt.parts.map((part) => part.text).join('')
|
||||
expect(excerptText).toContain('Farmácia Popular')
|
||||
expect(row.semanticMatch).toBe(false)
|
||||
expect(row.matchKind).toBe('segment')
|
||||
expect(row.matchedTextSearch).toBe(true)
|
||||
expect(row.excerpt.parts.some((part) => part.highlighted)).toBe(true)
|
||||
})
|
||||
|
||||
it('does not claim a theme the where never used (word-only coincidence)', () => {
|
||||
const row = themeRow({
|
||||
text: 'O atendimento público é prioridade',
|
||||
searchText: 'o atendimento publico e prioridade',
|
||||
query: 'defesa do SUS',
|
||||
themeTerms: ['atendimento universal'],
|
||||
it('seeks the evidence start when known and omits the seek without one', () => {
|
||||
const window = semanticRow({
|
||||
text: 'Outra fala',
|
||||
query: 'tema',
|
||||
evidence: { text: 'Trecho de janela', startSeconds: null },
|
||||
})
|
||||
expect(window.watchHref).toBe('/campanha/comunicacao/acervo/1?q=tema')
|
||||
})
|
||||
|
||||
it('flags a degraded literal row without claiming the theme', () => {
|
||||
const row = toSpeechListItemViewModel({
|
||||
speech: { id: 1, speechAt: '2026-08-11T18:48', searchText: 'a defesa do sus' },
|
||||
segments: [segment('A defesa do SUS')],
|
||||
query: 'SUS',
|
||||
municipalityLabels: new Map(),
|
||||
literalFallback: true,
|
||||
})
|
||||
|
||||
expect(row.themeMatchTerm).toBeNull()
|
||||
expect(row.matchKind).toBe('fallback')
|
||||
})
|
||||
|
||||
it('stays literal without theme terms or when none matches', () => {
|
||||
expect(
|
||||
themeRow({ text: 'A defesa do SUS', query: 'SUS', themeTerms: [] }).themeMatchTerm,
|
||||
).toBeNull()
|
||||
expect(
|
||||
themeRow({ text: 'A defesa do SUS', query: 'SUS', themeTerms: ['educação'] }).themeMatchTerm,
|
||||
).toBeNull()
|
||||
})
|
||||
|
||||
it('never claims a theme term the where mirror did not match (C192)', () => {
|
||||
// "saúde pública" never appears contiguously in the search text, so the
|
||||
// mirror rejects it and only the term that really matched is claimed.
|
||||
expect(
|
||||
themeRow({
|
||||
text: 'Saúde para todos e a rede pública de atendimento',
|
||||
searchText: 'saude para todos e a rede publica de atendimento',
|
||||
query: 'defesa do SUS',
|
||||
themeTerms: ['saúde pública', 'atendimento'],
|
||||
}).themeMatchTerm,
|
||||
).toBe('atendimento')
|
||||
})
|
||||
|
||||
it('falls back to keywords as theme evidence when the text does not match', () => {
|
||||
const row = themeRow({
|
||||
text: 'Uma fala sobre outro assunto',
|
||||
query: 'defesa do SUS',
|
||||
themeTerms: ['Farmácia Popular'],
|
||||
keywords: ['Farmácia Popular'],
|
||||
})
|
||||
|
||||
expect(row.themeMatchTerm).toBe('Farmácia Popular')
|
||||
expect(row.literalFallback).toBe(true)
|
||||
expect(row.semanticMatch).toBe(false)
|
||||
expect(row.matchKind).toBe('segment')
|
||||
expect(row.excerpt.parts.some((part) => part.highlighted)).toBe(true)
|
||||
})
|
||||
})
|
||||
|
|
@ -198,7 +183,6 @@ describe('web speech view models (C216)', () => {
|
|||
},
|
||||
segments: [],
|
||||
query: undefined,
|
||||
themeTerms: [],
|
||||
})
|
||||
|
||||
it('labels the platform and the day-only publication date', () => {
|
||||
|
|
|
|||
Loading…
Reference in a new issue