From 722cd6cf947f98abf6d4300df27bb0390bc4cc6a Mon Sep 17 00:00:00 2001 From: Francisco Solla Date: Mon, 28 Sep 2026 19:15:57 -0300 Subject: [PATCH] =?UTF-8?q?fix(C199):=20upload=20de=20grava=C3=A7=C3=B5es?= =?UTF-8?q?=20fatiado=20em=2016=20MiB=20=E2=80=94=20escapa=20do=20413=20do?= =?UTF-8?q?=20edge?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit O diálogo "Enviar gravação" mandava o arquivo inteiro num único XHR e a produção vive atrás do Cloudflare Tunnel, cujo edge responde 413 para qualquer request body acima de ~100 MB ANTES do app ver o corpo (probe read-only: POST de 150 MB cortado após ~3,2 MB). Um .mp4 de 6 GB de plenária nunca chegava à rota — o C199-large havia removido só o teto do app, não o do proxy. O upload agora é fatiado ponta a ponta: o start bodyless cria a row e a sessão em disco (`recording-upload-` com estado + arquivo anexado), o diálogo envia partes de 16 MiB em ordem estrita com progresso agregado e o último byte do tamanho declarado finaliza a mídia privada e o job. Cada parte faz heartbeat do `updatedAt` contra o reaper de 1 h, o start faz preflight de disco (statfs + margem), falha de parte aborta a sessão e o delete do usuário/reaper limpam o diretório. Sem migration e sem schema. Regressão: protocolo int nas rotas start/chunk/abort (multi-parte, fora de ordem, tamanho excedido, seam multipart, falha S3, reaper), guarda unit do orçamento do edge, guardas e2e das rotas. Post-mortem e entrada de changelog em docs/. --- ...-09-28-upload-gravacoes-teto-cloudflare.md | 1 + ...-09-28-upload-gravacoes-teto-cloudflare.md | 90 +++++ .../acervo/gravacoes/enviar/[id]/route.ts | 186 ++++++++++ .../acervo/gravacoes/enviar/route.ts | 44 ++- .../comunicacao/acervo/gravacoes/types.ts | 28 +- .../(campaign)/campanha/actions/recording.ts | 5 + .../recording/RecordingUploadDialog.tsx | 162 ++++++-- src/lib/campaignPaths.ts | 4 + src/lib/recording.ts | 36 ++ src/lib/schemas/recording.ts | 35 +- src/utilities/recordings/recordingJob.ts | 6 + src/utilities/recordings/recordingUpload.ts | 346 ++++++++++++++---- .../recordings/recordingUploadSession.ts | 99 +++++ tests/e2e/campaignSpeechAcervo.e2e.spec.ts | 9 +- tests/int/recording.int.spec.ts | 328 +++++++++++++++-- tests/unit/codebaseConventions.unit.spec.ts | 6 +- tests/unit/recording.unit.spec.ts | 24 ++ 17 files changed, 1250 insertions(+), 159 deletions(-) create mode 100644 docs/changelog/2026-09-28-upload-gravacoes-teto-cloudflare.md create mode 100644 docs/postmortems/2026-09-28-upload-gravacoes-teto-cloudflare.md create mode 100644 src/app/(campaign)/campanha/(app)/comunicacao/acervo/gravacoes/enviar/[id]/route.ts create mode 100644 src/utilities/recordings/recordingUploadSession.ts diff --git a/docs/changelog/2026-09-28-upload-gravacoes-teto-cloudflare.md b/docs/changelog/2026-09-28-upload-gravacoes-teto-cloudflare.md new file mode 100644 index 00000000..9206449a --- /dev/null +++ b/docs/changelog/2026-09-28-upload-gravacoes-teto-cloudflare.md @@ -0,0 +1 @@ +**Upload de gravações fatiado em 16 MiB escapa do teto de ~100 MB do Cloudflare (2026-09-28):** subir o `.mp4` de 6 GB de uma plenária no acervo de comunicação falhava porque o diálogo enviava o arquivo inteiro em um único XHR e o edge do Cloudflare respondia `413` antes do app ver o corpo — a recusa foi comprovada em produção por um probe read-only de 150 MB (corte após ~3,2 MB), não apurada na suíte, que nunca atravessa o edge. O upload agora é fatiado ponta a ponta com `RECORDING_UPLOAD_CHUNK_BYTES = 16 MiB` e `RECORDING_UPLOAD_EDGE_LIMIT_BYTES = 100 MiB`: o `POST .../gravacoes/enviar` vira start bodyless que devolve o `chunkSize` e abre a sessão em disco, `POST/DELETE .../gravacoes/enviar/[id]` recebem a parte crua por `?index=` e abortam de forma idempotente, e o diálogo fatia por `file.slice` com progresso agregado, aborto por `DELETE` com `keepalive` em erro/unmount e o reaper/`deleteRecordingForActor` limpando o dir da sessão. O start faz preflight de disco (`statfs(tmpdir())` com margem de 512 MiB) para não morrer de ENOSPC após horas, o `413` vira copy honesta no diálogo, e um unit pina `chunk < edge limit` (todo o resto acima de ~100 MB — inclusive a Central de Conteúdos — segue documentado como prevenção cara). Testes int do protocolo (multi-parte, fora de ordem, tamanho excedido, falha S3, reaper, rotas idempotentes) e guardas e2e das rotas verdes, `pnpm gate:fast`/`build`/`knip`/`cycles`/`format` verdes, sem migration e sem schema; PR #— aberto neste worktree, CI e confirmação em produção pendentes. diff --git a/docs/postmortems/2026-09-28-upload-gravacoes-teto-cloudflare.md b/docs/postmortems/2026-09-28-upload-gravacoes-teto-cloudflare.md new file mode 100644 index 00000000..53ddd411 --- /dev/null +++ b/docs/postmortems/2026-09-28-upload-gravacoes-teto-cloudflare.md @@ -0,0 +1,90 @@ +# Post-mortem: upload de gravações recusado pelo teto do edge Cloudflare + +> Template do `/bug-fix`. Preencha com fatos apurados; o que não for apurado fica "não apurado" — nunca invente. + +## Registro + +| Campo | Valor | +| ------------------- | --------------------------------------------------------------------------------------- | +| Data do post-mortem | 2026-09-28 | +| Severidade | alta (bloqueia subir gravações reais de horas no acervo de comunicação em produção) | +| Ambiente | produção (sintoma e diagnóstico read-only) + worktree `fix/14` (correção e verificação) | +| Issue(s) | sem Issue (relato humano direto no `/bug-fix`) | +| PR do fix | #— (aberto neste worktree) | +| Detectado por | humano (operador da campanha) | + +## Timeline + +| Momento | Data/hora | Evento | +| ------------------ | ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Início provável | 2026-09-18 | O C199 (gravações enviadas no acervo, commit d0f606a5) entrou com upload de arquivo inteiro em um único request; qualquer upload acima de ~100 MB já era recusado pelo edge do Cloudflare. O C199-large (remoção do teto de 4 GiB, commit 57b0d168, deploy verde em 2026-09-26 17:40) não mudou isso. | +| Detecção | 2026-09-28, hora não apurada | O humano relatou que subir um `.mp4` de 6 GB em `/campanha › Acervo › Gravações › "Enviar gravação"` "está falhando"; o diálogo mostra erro genérico e a mensagem exata na tela é não apurada. | +| Diagnóstico | 2026-09-28 | Reprodução read-only em produção: `curl -sSI https://jorgesolla1313.com.br/` → `server: cloudflare`; `POST` de 150 MB para um caminho inexistente (`/__cf-upload-limit-probe-does-not-exist`, não escreve nada) → `413 Payload Too Large` da Cloudflare, encerrando após ~3,2 MB enviados. | +| Correção mergeada | pendente | Correção implementada e verificada localmente; PR aberto neste worktree, CI pendente. | +| Deploy | pendente | O merge dispara o deploy; produção depende de approve humano no environment `production`. | +| Verificado em prod | pendente — confirmação humana | Pendente a confirmação do humano de que o upload de 6 GB funciona em produção. | + +## O bug + +Em produção, em `/campanha › Acervo › Gravações › "Enviar gravação"`, o envio de um `.mp4` de 6 GB falhava. A mensagem exata exibida na tela é não apurada; o diálogo mostra um erro genérico. O arquivo real de 6 GB não foi reproduzido localmente (não se faz upload de 6 GB em dev sem S3) — a recusa foi demonstrada de forma read-only em produção pelo probe de 150 MB, que o edge corta em `413` após ~3,2 MB. + +## Causa-raiz + +O diálogo enviava o arquivo INTEIRO em um único XHR (`xhr.send(file)`). Esse request único morre no edge do Cloudflare (tunnel de produção), que responde `413` antes do app ver o corpo. O único teto modelado era do app — o antigo `RECORDING_MAX_BYTES = 4 GiB`, removido no C199-large — e o limite do edge nunca foi nomeado nem parametrizado. Nenhum teste da suíte atravessa o edge (unit/int/e2e batem em localhost) e não existia invariante de "corpo por request < teto do proxy". + +Origem `file:line` pré-fix: + +- `src/components/campaign/recording/RecordingUploadDialog.tsx:164` (`xhr.send(file)`). +- `src/app/(campaign)/campanha/(app)/comunicacao/acervo/gravacoes/enviar/route.ts:85` (`body: request.body`). +- `src/utilities/recordings/recordingUpload.ts` (stream único, sem teto). + +### 5 whys + +1. Por que o upload de 6 GB falha? Porque o diálogo enviava o arquivo inteiro em um único XHR (`xhr.send(file)`). +2. Por que o request único falha? Porque ele morre no edge do Cloudflare, que responde `413 Payload Too Large` antes do app ver o corpo. +3. Por que o limite do edge não era tratado? Porque o único teto modelado era do app (o antigo `RECORDING_MAX_BYTES = 4 GiB`, removido no C199-large); o limite do edge nunca foi nomeado nem parametrizado. +4. Por que a suíte não pegou? Porque nenhum teste atravessa o edge (unit/int/e2e batem em localhost) e não existia invariante de "corpo por request < teto do proxy". +5. Por que isso passou? Sistêmico: limites de proxy/edge ficaram fora do orçamento de request do produto, e a validação de arquivos multi-GB foi por tipo/declaração — produção virou o único teste real. + +## Correção + +Upload fatiado ponta a ponta. `src/lib/recording.ts` ganhou `RECORDING_UPLOAD_CHUNK_BYTES = 16 MiB` e `RECORDING_UPLOAD_EDGE_LIMIT_BYTES = 100 MiB`. + +`src/utilities/recordings/recordingUploadSession.ts` (novo) passou a ser o dono do estado da sessão em disco: `os.tmpdir()/recording-upload-` com um `.session.json` de `{uploadName, expectedBytes}` mais o arquivo anexado e um `.placeholder/` interno. + +`src/utilities/recordings/recordingUpload.ts` foi refatorado em três operações: `startRecordingUpload` (cria a linha `uploading` e a sessão após o preflight de disco), `receiveRecordingChunk` (anexa a parte exigindo `index === floor(receivedBytes / chunkSize)`, aplica o teto por request, faz heartbeat do `updatedAt` contra o reaper de 1 h e finaliza a mídia privada + job quando os bytes recebidos igualam o tamanho declarado) e `abortRecordingUpload` (condicional, só a linha ainda `uploading`). + +As rotas ficaram `POST .../gravacoes/enviar` (bodyless, devolve `chunkSize`) e `POST/DELETE .../gravacoes/enviar/[id]` (parte crua com `?index=`, abort idempotente). O diálogo fatia com `file.slice` e envia as partes em ordem, com progresso agregado e aborto da sessão em erro/unmount (DELETE com `keepalive`). O reaper (`src/utilities/recordings/recordingJob.ts`) e `deleteRecordingForActor` removem o dir da sessão. Sem migration e sem mudança de schema. + +## Verificação + +- Teste de regressão: protocolo de upload fatiado em `tests/int/recording.int.spec.ts` (multi-parte, fora de ordem, parte grande, tamanho excedido, seam multipart, falha S3, reaper, start/chunk/abort idempotente); unit pina `RECORDING_UPLOAD_CHUNK_BYTES < RECORDING_UPLOAD_EDGE_LIMIT_BYTES`. +- Suíte: `pnpm gate:fast` verde (lint + typecheck + 4.943 unit em 448 arquivos); `pnpm test:int tests/int/recording.int.spec.ts` 31/31; `pnpm test:e2e campaignSpeechAcervo.e2e.spec.ts -g "upload routes refuse"` 3 passed (2 setup + 1 alvo); `pnpm build` verde; `pnpm knip`, `pnpm check:cycles` e `pnpm format:check` verdes. +- Verificador independente (sub-agente): reconfirmou unit 14/14, int 31/31, e2e 3 passed e o `413` read-only de produção; a caça a lacunas achou 4 pontos menores, dos quais 1 vazamento real foi corrigido (dir órfão quando a criação da sessão falha no start) e os outros 3 são corridas/limpezas best-effort documentadas. +- CI: pendente no PR. +- Prod: pendente de approve humano no environment `production` e da confirmação do humano de que o upload de 6 GB funciona. + +## Prevenção + +| Estratégia | Custo | Estado | +| --------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------ | ------------------------------------------ | +| Unit pinando `RECORDING_UPLOAD_CHUNK_BYTES < RECORDING_UPLOAD_EDGE_LIMIT_BYTES` (aumento acima do teto do edge quebra o build) | barata | implementada neste PR | +| Testes int do protocolo fatiado (multi-parte, fora de ordem, parte grande, tamanho excedido, seam multipart, falha S3, reaper, rotas idempotentes) | barata | implementada neste PR | +| Guardas e2e cross-origin/advisor nas duas rotas | barata | implementada neste PR | +| Preflight de disco no start (`statfs(tmpdir())` com margem de 512 MiB, via `recordingUploadFitsInDisk`, unit-testado) | barata | implementada neste PR | +| Mapeamento do `413` no diálogo para copy honesta (`RECORDING_UPLOAD_REJECTED_MESSAGE`) | barata | implementada neste PR | +| Retomada de upload (resume) por parte | cara | documentada — não implementada neste fluxo | +| Upload direto ao Garage S3 por URL pré-assinada (corpo sai do edge) | cara | documentada — não implementada neste fluxo | +| Observabilidade/alerta de `413` no edge (Cloudflare analytics) e smoke de >100 MB em staging atrás do tunnel | cara | documentada — não implementada neste fluxo | +| Fatiar também a Central de Conteúdos (`src/utilities/content/contentPieceUpload.ts`, `ContentPieceUploadDialog.tsx:158`, `ContentPieceAttachFileButton.tsx:72`) | cara | documentada — não implementada neste fluxo | + +**Estratégia implementada:** o corpo nunca mais atravessa o edge em um request único — o diálogo fatia em 16 MiB, o teto do edge (100 MiB) virou constante nomeada com teste que impede o chunk de crescer acima dele, o protocolo fatiado tem cobertura int/e2e, o start faz preflight de disco antes de horas de upload e o `413` do edge vira copy honesta no diálogo. + +**Estratégia documentada (cara):** resume por parte; upload direto ao Garage por URL pré-assinada; observabilidade de `413` no edge com smoke de >100 MB em staging atrás do tunnel; e o mesmo fatiamento na Central de Conteúdos, que sofre do MESMO `413` acima de ~100 MB e tem teto de produto de 4 GiB (`src/lib/contentPiece.ts:159`). Anexos de demanda (10 MB) e avatar (2 MB) estão abaixo do teto do edge e não são afetados. + +## Lições + +- A suíte inteira roda em localhost e nunca atravessou o edge — o limite do proxy não era fronteira nomeada no produto. +- A remoção do teto do app no C199-large deu a falsa sensação de que "qualquer tamanho passa"; o teto do edge sempre esteve lá, invisível. +- O diálogo tinha um erro genérico que escondia a recusa do edge — o app nem via o request, então nenhuma mensagem do app podia ser verdadeira. +- A feature nasceu com o caso real de "horas de plenária", mas o único teste de tamanho era o cap declarado, nunca o caminho real de bytes. diff --git a/src/app/(campaign)/campanha/(app)/comunicacao/acervo/gravacoes/enviar/[id]/route.ts b/src/app/(campaign)/campanha/(app)/comunicacao/acervo/gravacoes/enviar/[id]/route.ts new file mode 100644 index 00000000..dcf66a89 --- /dev/null +++ b/src/app/(campaign)/campanha/(app)/comunicacao/acervo/gravacoes/enviar/[id]/route.ts @@ -0,0 +1,186 @@ +import config from '@payload-config' +import { NextResponse } from 'next/server' +import { getPayload } from 'payload' + +import { canReadCommunicationCatalog } from '@/lib/campaignRoles' +import { + RECORDING_FILE_TYPE_MESSAGE, + recordingFileTypeAllowed, + toRecordingViewModel, +} from '@/lib/recording' +import { + RECORDING_CHUNK_OUT_OF_ORDER_MESSAGE, + RECORDING_CHUNK_TOO_LARGE_MESSAGE, + RECORDING_FORBIDDEN_MESSAGE, + RECORDING_GENERIC_ERROR_MESSAGE, + RECORDING_NOT_FOUND_MESSAGE, + RECORDING_UPLOAD_NOT_IN_PROGRESS_MESSAGE, + RECORDING_UPLOAD_SESSION_LOST_MESSAGE, + RECORDING_UPLOAD_SIZE_MISMATCH_MESSAGE, +} from '@/lib/schemas/recording' +import { getCampaignUser } from '@/utilities/campaignAuth' +import { CAMPAIGN_AUTH_REQUIRED_MESSAGE } from '@/utilities/campaignFormActionError' +import { campaignJsonMutationErrorResponse } from '@/utilities/campaignJsonMutationRoute' +import { strictDecimalInteger } from '@/utilities/campaignListUrl' +import { + abortRecordingUpload, + receiveRecordingChunk, + RECORDING_BODY_MISSING_MESSAGE, +} from '@/utilities/recordings/recordingUpload' +import { isSameOriginRequest } from '@/utilities/sameOriginRequest' + +import type { RecordingChunkResponse, RecordingUploadAbortResponse } from '../../types' + +type RouteContext = { + params: Promise<{ id: string }> +} + +export const dynamic = 'force-dynamic' + +/** + * C199-fix — receives one part of an in-flight recording upload as a RAW body + * (the request body IS the chunk, the index travels in the query). Sibling of + * the start route and the second half of the boundary workaround: production + * sits behind the Cloudflare Tunnel, whose edge refuses single request bodies + * above ~100 MB with a 413 before the app sees them, so a multi-GB recording + * is sliced by the dialog into parts of `RECORDING_UPLOAD_CHUNK_BYTES`. + * + * It cannot ride `campaignJsonMutationRoute` (JSON-body wrapper), so it repeats + * the same-origin guard and the error envelope explicitly and is allowlisted in + * the `codebaseConventions` sweep with its sibling. + */ + +const errorResponse = (message: string, status: number): NextResponse => + NextResponse.json({ status: 'error', message }, { status }) + +const SAFE_MESSAGES = [ + RECORDING_FORBIDDEN_MESSAGE, + RECORDING_NOT_FOUND_MESSAGE, + RECORDING_UPLOAD_NOT_IN_PROGRESS_MESSAGE, + RECORDING_UPLOAD_SESSION_LOST_MESSAGE, + RECORDING_CHUNK_OUT_OF_ORDER_MESSAGE, + RECORDING_CHUNK_TOO_LARGE_MESSAGE, + RECORDING_UPLOAD_SIZE_MISMATCH_MESSAGE, + RECORDING_FILE_TYPE_MESSAGE, + RECORDING_BODY_MISSING_MESSAGE, +] + +export const POST = async ( + request: Request, + context: RouteContext, +): Promise> => { + if (!isSameOriginRequest(request)) { + return errorResponse('Requisição inválida.', 403) + } + + const user = await getCampaignUser() + if (!user) return errorResponse(CAMPAIGN_AUTH_REQUIRED_MESSAGE, 401) + if (!canReadCommunicationCatalog(user.role)) { + return errorResponse(RECORDING_FORBIDDEN_MESSAGE, 403) + } + + const { id } = await context.params + const recordingId = strictDecimalInteger(id) + if (!recordingId) { + return errorResponse(RECORDING_NOT_FOUND_MESSAGE, 400) + } + + // `strictDecimalInteger` refuses 0, and the first part IS index 0. + const rawIndex = new URL(request.url).searchParams.get('index') ?? '' + const index = /^\d{1,9}$/.test(rawIndex) ? Number(rawIndex) : undefined + if (index === undefined) { + return errorResponse(RECORDING_CHUNK_OUT_OF_ORDER_MESSAGE, 400) + } + + if (!recordingFileTypeAllowed(request.headers.get('content-type'))) { + return errorResponse(RECORDING_FILE_TYPE_MESSAGE, 400) + } + + try { + const payload = await getPayload({ config }) + const { done } = await receiveRecordingChunk({ + payload, + actor: user, + recordingId, + index, + body: request.body, + }) + if (!done) { + return NextResponse.json({ status: 'success', done: false }) + } + + const recording = await payload.findByID({ + collection: 'recording', + id: recordingId, + depth: 0, + select: { + title: true, + status: true, + step: true, + recordedAt: true, + durationSeconds: true, + }, + user, + overrideAccess: false, + }) + return NextResponse.json({ + status: 'success', + done: true, + recording: toRecordingViewModel(recording), + }) + } catch (error) { + return campaignJsonMutationErrorResponse(error, { + safeMessages: SAFE_MESSAGES, + genericMessage: RECORDING_GENERIC_ERROR_MESSAGE, + }) + } +} + +/** + * C199-fix — aborts an in-flight upload: removes the temp session and the row, + * but ONLY while the row is still `uploading`. Idempotent on purpose: the + * dialog calls it when a part fails and a lost response may have finalized the + * upload already — the finalized recording must survive. + */ +export const DELETE = async ( + request: Request, + context: RouteContext, +): Promise> => { + if (!isSameOriginRequest(request)) { + return NextResponse.json({ status: 'error', message: 'Requisição inválida.' }, { status: 403 }) + } + + const user = await getCampaignUser() + if (!user) { + return NextResponse.json( + { status: 'error', message: CAMPAIGN_AUTH_REQUIRED_MESSAGE }, + { status: 401 }, + ) + } + if (!canReadCommunicationCatalog(user.role)) { + return NextResponse.json( + { status: 'error', message: RECORDING_FORBIDDEN_MESSAGE }, + { status: 403 }, + ) + } + + const { id } = await context.params + const recordingId = strictDecimalInteger(id) + if (!recordingId) { + return NextResponse.json( + { status: 'error', message: RECORDING_NOT_FOUND_MESSAGE }, + { status: 400 }, + ) + } + + try { + const payload = await getPayload({ config }) + await abortRecordingUpload(payload, recordingId) + return NextResponse.json({ status: 'success', aborted: true }) + } catch (error) { + return campaignJsonMutationErrorResponse(error, { + safeMessages: SAFE_MESSAGES, + genericMessage: RECORDING_GENERIC_ERROR_MESSAGE, + }) + } +} diff --git a/src/app/(campaign)/campanha/(app)/comunicacao/acervo/gravacoes/enviar/route.ts b/src/app/(campaign)/campanha/(app)/comunicacao/acervo/gravacoes/enviar/route.ts index b9b7cd26..f299f14e 100644 --- a/src/app/(campaign)/campanha/(app)/comunicacao/acervo/gravacoes/enviar/route.ts +++ b/src/app/(campaign)/campanha/(app)/comunicacao/acervo/gravacoes/enviar/route.ts @@ -4,54 +4,55 @@ import { getPayload } from 'payload' import { canReadCommunicationCatalog } from '@/lib/campaignRoles' import { - RECORDING_FILE_TYPE_MESSAGE, RECORDING_TITLE_REQUIRED_MESSAGE, - recordingFileTypeAllowed, + RECORDING_UPLOAD_CHUNK_BYTES, toRecordingViewModel, } from '@/lib/recording' import { RECORDING_FILE_NAME_MESSAGE, RECORDING_FORBIDDEN_MESSAGE, RECORDING_GENERIC_ERROR_MESSAGE, + RECORDING_UPLOAD_NO_SPACE_MESSAGE, recordingUploadMetadataSchema, } from '@/lib/schemas/recording' import { getCampaignUser } from '@/utilities/campaignAuth' import { CAMPAIGN_AUTH_REQUIRED_MESSAGE } from '@/utilities/campaignFormActionError' import { campaignJsonMutationErrorResponse } from '@/utilities/campaignJsonMutationRoute' -import { - receiveRecordingUpload, - RECORDING_BODY_MISSING_MESSAGE, -} from '@/utilities/recordings/recordingUpload' +import { startRecordingUpload } from '@/utilities/recordings/recordingUpload' import { isSameOriginRequest } from '@/utilities/sameOriginRequest' -import type { RecordingUploadResponse } from '../types' +import type { RecordingUploadStartResponse } from '../types' export const dynamic = 'force-dynamic' /** - * C199 — accepts one recording upload as a RAW body: the request body IS the - * video, the metadata travels in the query string. A route handler instead of a - * server action because a multi-GB file must stream to disk (the action default - * of 1 MB and the FormData buffer of hours of video are both unacceptable); XHR - * still gives the dialog a real `upload.onprogress`. + * C199-fix — starts one recording upload. The production edge (Cloudflare + * Tunnel) refuses any request body above ~100 MB with a 413 before the app + * sees it, so the file cannot travel in one request: this bodyless call + * creates the visible row and the on-disk session, then the dialog POSTs the + * file in parts to `.../gravacoes/enviar/[id]`. * * It cannot ride `campaignJsonMutationRoute` (JSON-body wrapper), so it repeats * the same-origin guard and the error envelope explicitly and is allowlisted in * the `codebaseConventions` sweep, like the multipart `ai-transcribe` route. */ -const errorResponse = (message: string, status: number): NextResponse => +const errorResponse = ( + message: string, + status: number, +): NextResponse => NextResponse.json({ status: 'error', message }, { status }) const SAFE_MESSAGES = [ RECORDING_FORBIDDEN_MESSAGE, RECORDING_TITLE_REQUIRED_MESSAGE, RECORDING_FILE_NAME_MESSAGE, - RECORDING_FILE_TYPE_MESSAGE, - RECORDING_BODY_MISSING_MESSAGE, + RECORDING_UPLOAD_NO_SPACE_MESSAGE, ] -export const POST = async (request: Request): Promise> => { +export const POST = async ( + request: Request, +): Promise> => { if (!isSameOriginRequest(request)) { return errorResponse('Requisição inválida.', 403) } @@ -67,22 +68,18 @@ export const POST = async (request: Request): Promise({ + return NextResponse.json({ status: 'success', recording: toRecordingViewModel(recording), + chunkSize: RECORDING_UPLOAD_CHUNK_BYTES, }) } catch (error) { return campaignJsonMutationErrorResponse(error, { diff --git a/src/app/(campaign)/campanha/(app)/comunicacao/acervo/gravacoes/types.ts b/src/app/(campaign)/campanha/(app)/comunicacao/acervo/gravacoes/types.ts index defb74bc..75ed3e44 100644 --- a/src/app/(campaign)/campanha/(app)/comunicacao/acervo/gravacoes/types.ts +++ b/src/app/(campaign)/campanha/(app)/comunicacao/acervo/gravacoes/types.ts @@ -1,8 +1,30 @@ import type { RecordingViewModel } from '@/lib/recording' -/** Wire contract of `POST .../gravacoes/enviar` (C199) — raw-body upload. */ -export type RecordingUploadResponse = - | { status: 'success'; recording: RecordingViewModel } +/** + * Wire contract of `POST .../gravacoes/enviar` (C199-fix) — starts a chunked + * upload and returns the part size the dialog must slice with. + */ +export type RecordingUploadStartResponse = + | { status: 'success'; recording: RecordingViewModel; chunkSize: number } + | { status: 'error'; message: string } + +/** + * Wire contract of `POST .../gravacoes/enviar/[id]?index=N` (C199-fix) — one + * raw-body part; the part that completes the declared size finalizes the + * upload and carries the recording. + */ +export type RecordingChunkResponse = + | { status: 'success'; done: false } + | { status: 'success'; done: true; recording: RecordingViewModel } + | { status: 'error'; message: string } + +/** + * Wire contract of `DELETE .../gravacoes/enviar/[id]` (C199-fix) — aborts an + * in-flight upload (temp session + `uploading` row); idempotent, a finalized + * recording is never touched. + */ +export type RecordingUploadAbortResponse = + | { status: 'success'; aborted: true } | { status: 'error'; message: string } /** Wire contract of `POST .../gravacoes/status` (C199) — status poll. */ diff --git a/src/app/(campaign)/campanha/actions/recording.ts b/src/app/(campaign)/campanha/actions/recording.ts index 755ee7e9..fab16085 100644 --- a/src/app/(campaign)/campanha/actions/recording.ts +++ b/src/app/(campaign)/campanha/actions/recording.ts @@ -25,6 +25,7 @@ import { onPayloadTransactionCommit, withPayloadTransaction } from '@/utilities/ import { acquireTextAdvisoryLocks } from '@/utilities/postgresTransactionLocks' import { reapStaleRecording } from '@/utilities/recordings/recordingJob' import { startRecordingJobInBackground } from '@/utilities/recordings/recordingScheduler' +import { removeRecordingUploadSession } from '@/utilities/recordings/recordingUploadSession' /** * C199 — mutations of one uploaded recording: retry the failed transcription, @@ -266,5 +267,9 @@ export const deleteRecordingForActor = async (input: { } }) + // C199-fix — a row deleted mid-upload owns a temp session with the partial + // file; the row is gone and nothing else would ever clean it. + await removeRecordingUploadSession(parsed.recordingId) + return { deleted: true } } diff --git a/src/components/campaign/recording/RecordingUploadDialog.tsx b/src/components/campaign/recording/RecordingUploadDialog.tsx index 4febb7eb..f01bc4e7 100644 --- a/src/components/campaign/recording/RecordingUploadDialog.tsx +++ b/src/components/campaign/recording/RecordingUploadDialog.tsx @@ -4,7 +4,10 @@ import { CircleAlertIcon, FileIcon, UploadIcon, VideoIcon } from 'lucide-react' import { useRouter } from 'next/navigation' import { useEffect, useRef, useState, type ChangeEvent } from 'react' -import type { RecordingUploadResponse } from '@/app/(campaign)/campanha/(app)/comunicacao/acervo/gravacoes/types' +import type { + RecordingChunkResponse, + RecordingUploadStartResponse, +} from '@/app/(campaign)/campanha/(app)/comunicacao/acervo/gravacoes/types' import { Alert, AlertDescription } from '@/components/ui/Alert' import { Button } from '@/components/ui/button' import { @@ -19,13 +22,17 @@ import { import { Input } from '@/components/ui/input' import { Label } from '@/components/ui/label' import { Spinner } from '@/components/ui/Spinner' -import { CAMPAIGN_RECORDING_UPLOAD_HREF } from '@/lib/campaignPaths' +import { + CAMPAIGN_RECORDING_UPLOAD_HREF, + campaignRecordingUploadSessionHref, +} from '@/lib/campaignPaths' import { formatRecordingFileSize, RECORDING_FILE_REQUIRED_MESSAGE, RECORDING_FILE_TYPE_MESSAGE, RECORDING_TITLE_MAX_LENGTH, RECORDING_TITLE_REQUIRED_MESSAGE, + RECORDING_UPLOAD_REJECTED_MESSAGE, recordingFileTypeAllowed, } from '@/lib/recording' import { cn } from '@/lib/utils' @@ -44,10 +51,12 @@ type RecordingUploadDialogProps = { } /** - * C199 — the one upload surface of the recordings source. Sends the RAW file - * with XHR (real `upload.onprogress`, no server-action buffer) with the metadata - * in the query string; the dialog stays open and unclosable while the file is - * in flight, per the approved design. + * C199/C199-fix — the one upload surface of the recordings source. Starts a + * session and sends the file in parts (the production edge refuses a single + * request body above ~100 MB with a 413 before the app sees it), with XHR for a + * real `upload.onprogress` per part and the metadata in the query string; the + * dialog stays open and unclosable while the file is in flight, per the + * approved design. */ export const RecordingUploadDialog = ({ triggerClassName, @@ -58,6 +67,7 @@ export const RecordingUploadDialog = ({ const router = useRouter() const fileInputRef = useRef(null) const xhrRef = useRef(null) + const uploadIdRef = useRef(null) const [open, setOpen] = useState(false) const [file, setFile] = useState(null) @@ -71,13 +81,29 @@ export const RecordingUploadDialog = ({ const uploading = phase === 'uploading' - // Navigating away unmounts the dialog: abort the in-flight XHR instead of - // leaving an invisible multi-GB upload running (the reaper cleans a row left - // in `uploading` by a crash). + // A failed or abandoned upload must not linger as a phantom "Enviando": the + // abort endpoint discards the row AND its temp session, and only while the + // row is still `uploading` — a final part whose response was lost keeps the + // finalized recording. Best-effort; the reaper backstops a crash, and + // `keepalive` lets the DELETE survive a navigation. + const discardSession = () => { + const uploadId = uploadIdRef.current + uploadIdRef.current = null + if (uploadId === null) return + void fetch(campaignRecordingUploadSessionHref(uploadId), { + method: 'DELETE', + headers: { Accept: 'application/json' }, + keepalive: true, + }).catch(() => undefined) + } + + // Navigating away unmounts the dialog: abort the in-flight part instead of + // leaving an invisible multi-GB upload running, and discard the session. useEffect( () => () => { xhrRef.current?.abort() xhrRef.current = null + discardSession() }, [], ) @@ -109,7 +135,21 @@ export const RecordingUploadDialog = ({ } } - const submit = () => { + const fail = (message: string) => { + setPhase('error') + setError(message) + discardSession() + } + + const parseJson = (text: string): T | null => { + try { + return JSON.parse(text) as T + } catch { + return null + } + } + + const submit = async () => { const nextTitle = title.trim() setTitleError(null) setError(null) @@ -123,45 +163,93 @@ export const RecordingUploadDialog = ({ setTitleError(RECORDING_TITLE_REQUIRED_MESSAGE) return } + const selectedFile = file - const params = new URLSearchParams({ title: nextTitle, filename: file.name }) + const params = new URLSearchParams({ + title: nextTitle, + filename: selectedFile.name, + size: String(selectedFile.size), + }) if (recordedAt) params.set('recordedAt', recordedAt) - const xhr = new XMLHttpRequest() - xhrRef.current = xhr setPhase('uploading') setProgress(0) - xhr.open('POST', `${CAMPAIGN_RECORDING_UPLOAD_HREF}?${params.toString()}`) - xhr.setRequestHeader('Accept', 'application/json') - xhr.upload.addEventListener('progress', (event) => { - if (event.lengthComputable && event.total > 0) { - setProgress(Math.round((event.loaded / event.total) * 100)) + // 1) Start the session: visible row + temp file the parts will grow. + let started: { id: number; chunkSize: number } + try { + const response = await fetch(`${CAMPAIGN_RECORDING_UPLOAD_HREF}?${params.toString()}`, { + method: 'POST', + headers: { Accept: 'application/json' }, + }) + const payload = parseJson(await response.text()) + if (!response.ok || !payload || payload.status !== 'success') { + fail(payload && payload.status === 'error' ? payload.message : GENERIC_ERROR_MESSAGE) + return } - }) - xhr.addEventListener('load', () => { - xhrRef.current = null - try { - const payload = JSON.parse(xhr.responseText) as RecordingUploadResponse - if (xhr.status >= 200 && xhr.status < 300 && payload.status === 'success') { + started = { id: payload.recording.id, chunkSize: payload.chunkSize } + } catch { + fail(GENERIC_ERROR_MESSAGE) + return + } + + uploadIdRef.current = started.id + const totalChunks = Math.max(1, Math.ceil(selectedFile.size / started.chunkSize)) + + // 2) Send the parts strictly in order; the last one finalizes the upload. + const sendChunk = (index: number) => { + const start = index * started.chunkSize + const chunk = selectedFile.slice( + start, + Math.min(start + started.chunkSize, selectedFile.size), + ) + const xhr = new XMLHttpRequest() + xhrRef.current = xhr + + xhr.open('POST', `${campaignRecordingUploadSessionHref(started.id)}?index=${index}`) + xhr.setRequestHeader('Accept', 'application/json') + xhr.upload.addEventListener('progress', (event) => { + if (selectedFile.size === 0) return + const loaded = start + (event.lengthComputable ? event.loaded : 0) + setProgress(Math.min(99, Math.round((loaded / selectedFile.size) * 100))) + }) + xhr.addEventListener('load', () => { + xhrRef.current = null + // A 413 comes from a proxy/edge, not from the app: the body never + // arrived and no JSON envelope exists. + if (xhr.status === 413) { + fail(RECORDING_UPLOAD_REJECTED_MESSAGE) + return + } + const payload = parseJson(xhr.responseText) + if (xhr.status < 200 || xhr.status >= 300 || !payload || payload.status !== 'success') { + fail(payload && payload.status === 'error' ? payload.message : GENERIC_ERROR_MESSAGE) + return + } + if (payload.done) { + uploadIdRef.current = null + setProgress(100) reset() setOpen(false) router.refresh() return } - setPhase('error') - setError(payload.status === 'error' ? payload.message : GENERIC_ERROR_MESSAGE) - } catch { - setPhase('error') - setError(GENERIC_ERROR_MESSAGE) - } - }) - xhr.addEventListener('error', () => { - xhrRef.current = null - setPhase('error') - setError(GENERIC_ERROR_MESSAGE) - }) - xhr.send(file) + if (index + 1 >= totalChunks) { + // The server did not reach the declared size: the file is not the + // same the dialog measured. + fail(GENERIC_ERROR_MESSAGE) + return + } + sendChunk(index + 1) + }) + xhr.addEventListener('error', () => { + xhrRef.current = null + fail(GENERIC_ERROR_MESSAGE) + }) + xhr.send(chunk) + } + + sendChunk(0) } return ( diff --git a/src/lib/campaignPaths.ts b/src/lib/campaignPaths.ts index 21a27a3d..fffde6ce 100644 --- a/src/lib/campaignPaths.ts +++ b/src/lib/campaignPaths.ts @@ -110,6 +110,10 @@ const campaignRecordingDetailHref = (id: number): string => export const CAMPAIGN_RECORDING_UPLOAD_HREF = `${CAMPAIGN_COMMUNICATION_ACERVO_GRAVACOES}/enviar` as const +/** C199-fix — one in-flight chunked upload session: POST a part, DELETE aborts. */ +export const campaignRecordingUploadSessionHref = (id: number): string => + `${CAMPAIGN_RECORDING_UPLOAD_HREF}/${id}` + export const CAMPAIGN_RECORDING_STATUS_HREF = `${CAMPAIGN_COMMUNICATION_ACERVO_GRAVACOES}/status` as const diff --git a/src/lib/recording.ts b/src/lib/recording.ts index bed63a57..eba52ba9 100644 --- a/src/lib/recording.ts +++ b/src/lib/recording.ts @@ -30,6 +30,42 @@ export const isRecordingStatus = (value: unknown): value is RecordingStatus => /** One owner for the persisted field limits (collection, schema and dialog). */ export const RECORDING_TITLE_MAX_LENGTH = 200 +/** + * Every request body that crosses the production edge (Cloudflare Tunnel) must + * stay under the edge's request ceiling — a bigger single request is refused + * with `413 Payload Too Large` BEFORE the app sees it (probed against prod: + * 150 MB answered 413 after ~3 MB). The dialog therefore slices the file into + * parts of this size; 16 MiB keeps a wide margin under the ~100 MB edge limit + * and still moves hours of video in few requests. + */ +export const RECORDING_UPLOAD_CHUNK_BYTES = 16 * 1024 * 1024 + +/** The edge ceiling the chunk size must stay under (Cloudflare: ~100 MB). */ +export const RECORDING_UPLOAD_EDGE_LIMIT_BYTES = 100 * 1024 * 1024 + +/** + * Free-space margin the upload start demands beyond the declared file size: + * the temp copy must fit with room for the post-upload job (which downloads + * the same file again) and the running container. Failing here is honest and + * immediate; failing mid-way wastes hours of transfer. + */ +const RECORDING_UPLOAD_DISK_MARGIN_BYTES = 512 * 1024 * 1024 + +/** Pure budget check of the upload start (the fs call lives in the session owner). */ +export const recordingUploadFitsInDisk = ({ + freeBytes, + expectedBytes, + marginBytes = RECORDING_UPLOAD_DISK_MARGIN_BYTES, +}: { + freeBytes: number + expectedBytes: number + marginBytes?: number +}): boolean => Number.isFinite(freeBytes) && freeBytes - expectedBytes >= marginBytes + +/** A proxy/edge refused the request body (413) before the app ever saw it. */ +export const RECORDING_UPLOAD_REJECTED_MESSAGE = + 'O servidor recusou uma parte do envio por tamanho. Tente novamente; se persistir, avise a equipe.' + /** C200 — the human label the team types for one speaker cluster. */ export const RECORDING_SPEAKER_LABEL_MAX_LENGTH = 120 export const RECORDING_SPEAKER_LABEL_REQUIRED_MESSAGE = 'Informe o nome do falante.' diff --git a/src/lib/schemas/recording.ts b/src/lib/schemas/recording.ts index e574776c..ae2821f4 100644 --- a/src/lib/schemas/recording.ts +++ b/src/lib/schemas/recording.ts @@ -26,6 +26,32 @@ export const RECORDING_GENERIC_ERROR_MESSAGE = /** The recording upload arrived without a file name. */ export const RECORDING_FILE_NAME_MESSAGE = 'Nome de arquivo inválido.' +/** The declared file size arrived missing or invalid. */ +const RECORDING_FILE_SIZE_MESSAGE = 'Tamanho de arquivo inválido.' + +/** C199-fix — one chunk arrived out of order (or duplicated). */ +export const RECORDING_CHUNK_OUT_OF_ORDER_MESSAGE = + 'Uma parte do envio chegou fora de ordem. Envie a gravação novamente.' + +/** C199-fix — one request carried more bytes than a chunk may carry. */ +export const RECORDING_CHUNK_TOO_LARGE_MESSAGE = 'Uma parte do envio excedeu o tamanho permitido.' + +/** C199-fix — the bytes received do not match the declared file size. */ +export const RECORDING_UPLOAD_SIZE_MISMATCH_MESSAGE = + 'O envio não corresponde ao tamanho do arquivo informado. Envie a gravação novamente.' + +/** C199-fix — the upload session state is gone (restart or manual cleanup). */ +export const RECORDING_UPLOAD_SESSION_LOST_MESSAGE = + 'O envio da gravação foi interrompido. Comece de novo.' + +/** C199-fix — chunks of a recording that is no longer uploading. */ +export const RECORDING_UPLOAD_NOT_IN_PROGRESS_MESSAGE = + 'Este envio de gravação não está mais em andamento.' + +/** C199-fix — the server has no disk room for the declared file plus margin. */ +export const RECORDING_UPLOAD_NO_SPACE_MESSAGE = + 'Espaço insuficiente no servidor para receber esta gravação. Avise a equipe.' + /** Filename sanity: no path parts, no empties — the temp file keeps the extension. */ const uploadFilename = z .string() @@ -34,7 +60,7 @@ const uploadFilename = z .max(255, RECORDING_FILE_NAME_MESSAGE) .refine((value) => !value.includes('/') && !value.includes('\\'), RECORDING_FILE_NAME_MESSAGE) -/** Metadata of the raw-body upload (`POST .../gravacoes/enviar`). */ +/** Metadata of the chunked upload (`POST .../gravacoes/enviar`). */ export const recordingUploadMetadataSchema = z.object({ title: z .string() @@ -46,6 +72,13 @@ export const recordingUploadMetadataSchema = z.object({ ), recordedAt: trimmedOptionalText(32), filename: uploadFilename, + /** Declared size of the whole file; the server finalizes only when it matches. */ + size: z + .string() + .trim() + .regex(/^\d+$/, RECORDING_FILE_SIZE_MESSAGE) + .transform(Number) + .refine((value) => Number.isSafeInteger(value) && value >= 0, RECORDING_FILE_SIZE_MESSAGE), }) export type RecordingUploadMetadata = z.infer diff --git a/src/utilities/recordings/recordingJob.ts b/src/utilities/recordings/recordingJob.ts index bb798eef..ee02a9ca 100644 --- a/src/utilities/recordings/recordingJob.ts +++ b/src/utilities/recordings/recordingJob.ts @@ -44,6 +44,7 @@ import { recordingFacetWriteData, type RecordingFacetClassifier, } from '@/utilities/recordings/recordingClassification' +import { removeRecordingUploadSession } from '@/utilities/recordings/recordingUploadSession' import { classifySpeech } from '@/utilities/speech/speechClassifier' /** @@ -366,6 +367,11 @@ export const reapStaleRecording = async ( // Intentional admin bypass: the reaper repairs a row the system owns. overrideAccess: true, }) + if (deleted.docs.length > 0) { + // C199-fix — the row owned a chunked upload session; without it the temp + // copy of a multi-GB recording would sit in the disk forever. + await removeRecordingUploadSession(recording.id) + } return deleted.docs.length > 0 } diff --git a/src/utilities/recordings/recordingUpload.ts b/src/utilities/recordings/recordingUpload.ts index 7fae3893..10c64188 100644 --- a/src/utilities/recordings/recordingUpload.ts +++ b/src/utilities/recordings/recordingUpload.ts @@ -1,72 +1,101 @@ import 'server-only' import { createWriteStream } from 'node:fs' -import { mkdtemp, rm } from 'node:fs/promises' -import { tmpdir } from 'node:os' -import { join } from 'node:path' -import { Readable } from 'node:stream' +import { stat } from 'node:fs/promises' +import { Readable, Transform } from 'node:stream' import { pipeline } from 'node:stream/promises' import type { Payload } from 'payload' -import { RECORDING_MEDIA_SLUG, sanitizeRecordingFilename } from '@/lib/recording' -import type { RecordingUploadMetadata } from '@/lib/schemas/recording' +import { + RECORDING_MEDIA_SLUG, + RECORDING_UPLOAD_CHUNK_BYTES, + sanitizeRecordingFilename, +} from '@/lib/recording' +import { + RECORDING_CHUNK_OUT_OF_ORDER_MESSAGE, + RECORDING_CHUNK_TOO_LARGE_MESSAGE, + RECORDING_NOT_FOUND_MESSAGE, + RECORDING_UPLOAD_NOT_IN_PROGRESS_MESSAGE, + RECORDING_UPLOAD_SESSION_LOST_MESSAGE, + RECORDING_UPLOAD_SIZE_MISMATCH_MESSAGE, + type RecordingUploadMetadata, +} from '@/lib/schemas/recording' import type { CampaignUser } from '@/payload-types' import { deleteLargeMedia, uploadLargeMedia } from '@/utilities/media/largeS3Upload' import { withPayloadTransaction } from '@/utilities/payloadTransaction' import { createPrivateMediaFromFile } from '@/utilities/privateMedia/privateMediaUpload' import { startRecordingJobInBackground } from '@/utilities/recordings/recordingScheduler' +import { + assertRecordingUploadDiskRoom, + createRecordingUploadSession, + readRecordingUploadSession, + recordingUploadFilePath, + recordingUploadPlaceholderPath, + removeRecordingUploadSession, + type RecordingUploadSession, +} from '@/utilities/recordings/recordingUploadSession' /** - * C199 — the upload half of the recordings flow: create the visible row, stream - * the raw request body to a temp file, then store the private media (multipart - * above the Payload buffer boundary, see `privateMediaUpload`) and hand the row - * to the transcription job. The route owns the origin/session/role gate; this - * module owns the write path. + * C199-fix — the upload half of the recordings flow, now chunked. The dialog + * starts a session (row `uploading` + temp dir), then POSTs the file in parts + * of `RECORDING_UPLOAD_CHUNK_BYTES`: production is behind the Cloudflare + * Tunnel, whose edge refuses any single request body above ~100 MB with a 413 + * before the app sees it, so a multi-GB recording MUST travel in parts. Each + * part is appended in strict order, finalize happens only when the received + * bytes match the size declared at start, and the private media + job are the + * same C199/C199-large flow as before. + * + * The route owns the origin/session/role gate; this module owns the write path. + * Any failed part aborts the session (row and temp dir), so a broken upload + * never lingers as a phantom "Enviando". */ /** Shared literal: the route allowlists it and the missing body throws it. */ export const RECORDING_BODY_MISSING_MESSAGE = 'A requisição não trouxe o arquivo da gravação.' -/** - * Streams the request body to `destination` with backpressure. There is no - * product ceiling (the upload accepts hours of plenary): the container disk and - * the S3 multipart limit are the bounds. - */ -const streamBodyToFile = async ({ - body, - destination, -}: { - body: ReadableStream - destination: string -}): Promise => { - await pipeline( - Readable.fromWeb(body as unknown as import('node:stream/web').ReadableStream), - createWriteStream(destination), - ) +const deleteUploadingRow = async (payload: Payload, recordingId: number): Promise => { + await payload + .delete({ + collection: 'recording', + where: { and: [{ id: { equals: recordingId } }, { status: { equals: 'uploading' } }] }, + // Intentional admin bypass: cleanup of the session this module owns. The + // conditional `where` never deletes a row another actor already finalized. + overrideAccess: true, + }) + .catch(() => undefined) } -export const receiveRecordingUpload = async ({ +/** + * Removes the row (only while it is still `uploading`) and the temp dir; the + * same outcome as a failed single-shot upload. Order matters on purpose: a + * concurrent finalize keeps its row until it commits, so the dir is only + * discarded after the conditional delete had a chance to lose that race. + */ +export const abortRecordingUpload = async ( + payload: Payload, + recordingId: number, +): Promise => { + await deleteUploadingRow(payload, recordingId) + await removeRecordingUploadSession(recordingId) +} + +/** + * Creates the visible row the chunk route will grow and the on-disk session. + * A failure while creating the session removes the row: an aborted start never + * leaves a phantom "Enviando". + */ +export const startRecordingUpload = async ({ payload, actor, metadata, - body, - startJob = startRecordingJobInBackground, - largeThresholdBytes, - uploadLarge, - removeLarge, }: { payload: Payload actor: CampaignUser metadata: RecordingUploadMetadata - body: ReadableStream | null - /** Injectable for tests; the route uses the `after()`-based default. */ - startJob?: (recordingId: number) => void - /** Test seams; the production defaults live in the private-media owner. */ - largeThresholdBytes?: number - uploadLarge?: typeof uploadLargeMedia - removeLarge?: typeof deleteLargeMedia }): Promise<{ id: number }> => { - if (!body) throw new Error(RECORDING_BODY_MISSING_MESSAGE) + // Fail before any write when the disk cannot hold the declared file: the + // user learns it in seconds, not after hours of transfer. + await assertRecordingUploadDiskRoom(metadata.size) const recording = await payload.create({ collection: 'recording', @@ -80,21 +109,114 @@ export const receiveRecordingUpload = async ({ overrideAccess: false, }) - let tempDir: string | null = null + try { + await createRecordingUploadSession(recording.id, { + uploadName: sanitizeRecordingFilename(metadata.filename), + expectedBytes: metadata.size, + }) + } catch (error) { + // A half-created session (dir without the empty file, or without the + // state) is unreachable by every other cleanup path: the row is gone and + // the reaper only looks at rows. + await abortRecordingUpload(payload, recording.id) + throw error + } + return { id: recording.id } +} + +/** + * Appends one chunk to the session file with backpressure, refusing a request + * that carries more than one chunk or that would cross the declared size. + * There is no product ceiling (the upload accepts hours of plenary): the + * container disk and the S3 multipart limit are the bounds. + */ +const appendChunkToFile = async ({ + body, + destination, + initialBytes, + expectedBytes, + chunkSize, +}: { + body: ReadableStream + destination: string + initialBytes: number + expectedBytes: number + chunkSize: number +}): Promise => { + let received = initialBytes + const guard = new Transform({ + transform(chunk: Buffer, _encoding, callback) { + received += chunk.length + if (received - initialBytes > chunkSize) { + callback(new Error(RECORDING_CHUNK_TOO_LARGE_MESSAGE)) + return + } + if (received > expectedBytes) { + callback(new Error(RECORDING_UPLOAD_SIZE_MISMATCH_MESSAGE)) + return + } + callback(null, chunk) + }, + }) + + await pipeline( + Readable.fromWeb(body as unknown as import('node:stream/web').ReadableStream), + guard, + createWriteStream(destination, { flags: 'a' }), + ) + return received +} + +/** Keeps `updatedAt` fresh while a multi-hour upload streams part by part. */ +const heartbeatRecordingUpload = ( + payload: Payload, + actor: CampaignUser, + recordingId: number, +): Promise => + payload.update({ + collection: 'recording', + id: recordingId, + data: { status: 'uploading' }, + depth: 0, + user: actor, + overrideAccess: false, + }) + +/** + * Stores the private media of a complete session and hands the row to the job. + * A rolled-back transaction leaves the multipart object (when the large path + * ran) with no row owning it, so it is removed too; the session dir goes away + * with the outer abort. + */ +const finalizeRecordingUpload = async ({ + payload, + actor, + recordingId, + title, + session, + startJob, + largeThresholdBytes, + uploadLarge, + removeLarge, +}: { + payload: Payload + actor: CampaignUser + recordingId: number + title: string + session: RecordingUploadSession + startJob: (recordingId: number) => void + largeThresholdBytes?: number + uploadLarge?: typeof uploadLargeMedia + removeLarge?: typeof deleteLargeMedia +}): Promise => { const mediaCleanup: { current?: () => Promise } = {} try { - const uploadDir = await mkdtemp(join(tmpdir(), 'recording-upload-')) - tempDir = uploadDir - const uploadName = sanitizeRecordingFilename(metadata.filename) - const tempPath = join(uploadDir, uploadName) - await streamBodyToFile({ body, destination: tempPath }) - await withPayloadTransaction(payload, async ({ req }) => { const media = await createPrivateMediaFromFile({ - inputPath: tempPath, + inputPath: recordingUploadFilePath(recordingId, session.uploadName), // Same basename as the upload: the stored name (and the download) keeps // the original file name even on the multipart path. - placeholderPath: join(uploadDir, 'placeholder', uploadName), + placeholderPath: recordingUploadPlaceholderPath(recordingId, session.uploadName), thresholdBytes: largeThresholdBytes, // No product ceiling for recordings; disk is the bound. maxBytes: Number.POSITIVE_INFINITY, @@ -103,7 +225,7 @@ export const receiveRecordingUpload = async ({ create: (filePath) => payload.create({ collection: RECORDING_MEDIA_SLUG, - data: { alt: metadata.title }, + data: { alt: title }, filePath, depth: 0, user: actor, @@ -125,7 +247,7 @@ export const receiveRecordingUpload = async ({ await payload.update({ collection: 'recording', - id: recording.id, + id: recordingId, data: { status: 'processing', step: 'extracting', media: media.id }, depth: 0, user: actor, @@ -136,23 +258,113 @@ export const receiveRecordingUpload = async ({ // Committed: the multipart object is the recording's media now. mediaCleanup.current = undefined - startJob(recording.id) - return { id: recording.id } + startJob(recordingId) } catch (error) { - // The upload never completed: the row must not linger as a phantom - // "Enviando". A rolled-back transaction leaves the multipart object (when - // the large path ran) with no row owning it, so it is removed too. await mediaCleanup.current?.().catch(() => undefined) - await payload - .delete({ - collection: 'recording', - id: recording.id, - // Intentional admin bypass: cleanup of the row this module just created. - overrideAccess: true, - }) - .catch(() => undefined) throw error - } finally { - if (tempDir) await rm(tempDir, { recursive: true, force: true }).catch(() => undefined) + } +} + +/** + * Receives one part of an in-flight upload. The index must be exactly the next + * part (`floor(receivedBytes / chunkSize)`), so a duplicated or out-of-order + * part is refused instead of corrupting the file; a chunk that completes the + * declared size finalizes the media in the same request. + */ +export const receiveRecordingChunk = async ({ + payload, + actor, + recordingId, + index, + body, + startJob = startRecordingJobInBackground, + largeThresholdBytes, + chunkSize = RECORDING_UPLOAD_CHUNK_BYTES, + uploadLarge, + removeLarge, +}: { + payload: Payload + actor: CampaignUser + recordingId: number + index: number + body: ReadableStream | null + /** Injectable for tests; the route uses the `after()`-based default. */ + startJob?: (recordingId: number) => void + /** Test seams; the production defaults live in the private-media owner. */ + largeThresholdBytes?: number + /** Part size the index math and the per-request guard agree on. */ + chunkSize?: number + uploadLarge?: typeof uploadLargeMedia + removeLarge?: typeof deleteLargeMedia +}): Promise<{ done: boolean }> => { + if (!body) throw new Error(RECORDING_BODY_MISSING_MESSAGE) + + const row = ( + await payload.find({ + collection: 'recording', + where: { id: { equals: recordingId } }, + depth: 0, + limit: 1, + pagination: false, + select: { status: true, title: true }, + user: actor, + overrideAccess: false, + }) + ).docs[0] + if (!row) throw new Error(RECORDING_NOT_FOUND_MESSAGE) + if (row.status !== 'uploading') throw new Error(RECORDING_UPLOAD_NOT_IN_PROGRESS_MESSAGE) + + const session = await readRecordingUploadSession(recordingId) + if (!session) { + await abortRecordingUpload(payload, recordingId) + throw new Error(RECORDING_UPLOAD_SESSION_LOST_MESSAGE) + } + + const destination = recordingUploadFilePath(recordingId, session.uploadName) + let receivedBefore: number + try { + receivedBefore = (await stat(destination)).size + } catch { + await abortRecordingUpload(payload, recordingId) + throw new Error(RECORDING_UPLOAD_SESSION_LOST_MESSAGE) + } + + if (index !== Math.floor(receivedBefore / chunkSize)) { + await abortRecordingUpload(payload, recordingId) + throw new Error(RECORDING_CHUNK_OUT_OF_ORDER_MESSAGE) + } + + try { + const received = await appendChunkToFile({ + body, + destination, + initialBytes: receivedBefore, + expectedBytes: session.expectedBytes, + chunkSize, + }) + + if (received < session.expectedBytes) { + await heartbeatRecordingUpload(payload, actor, recordingId) + return { done: false } + } + + await finalizeRecordingUpload({ + payload, + actor, + recordingId, + title: row.title ?? `Gravação ${recordingId}`, + session, + startJob, + largeThresholdBytes, + uploadLarge, + removeLarge, + }) + // The session file was fully consumed and committed: the temp copy never + // outlives the request that completed it. + await removeRecordingUploadSession(recordingId) + return { done: true } + } catch (error) { + await abortRecordingUpload(payload, recordingId) + throw error } } diff --git a/src/utilities/recordings/recordingUploadSession.ts b/src/utilities/recordings/recordingUploadSession.ts new file mode 100644 index 00000000..c2b30356 --- /dev/null +++ b/src/utilities/recordings/recordingUploadSession.ts @@ -0,0 +1,99 @@ +import 'server-only' + +import { mkdir, readFile, rm, statfs, writeFile } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' + +import { recordingUploadFitsInDisk } from '@/lib/recording' +import { RECORDING_UPLOAD_NO_SPACE_MESSAGE } from '@/lib/schemas/recording' + +/** + * C199-fix — on-disk state of one in-flight chunked recording upload: a + * per-recording directory under the temp dir holding the video appended chunk + * by chunk plus the session metadata (sanitized file name and the declared + * byte count). The recording row owns auth and visibility; this directory owns + * the bytes. + * + * Lives apart from `recordingUpload` and `recordingJob` so both the upload path + * and the reaper can create/read/remove a session without an import cycle. + * Internal names are dot-prefixed; `sanitizeRecordingFilename` strips leading + * dots, so a user file name can never collide with them. + */ + +const SESSION_STATE_FILE = '.session.json' +const PLACEHOLDER_DIR = '.placeholder' + +export type RecordingUploadSession = { + /** Sanitized original file name; the appended file (and the stored media) keeps it. */ + uploadName: string + /** Size the client declared; finalize only happens when it matches exactly. */ + expectedBytes: number +} + +type SessionRecord = RecordingUploadSession + +const recordingUploadDir = (recordingId: number): string => + join(tmpdir(), `recording-upload-${recordingId}`) + +export const recordingUploadFilePath = (recordingId: number, uploadName: string): string => + join(recordingUploadDir(recordingId), uploadName) + +/** Dot-prefixed placeholder subdirectory; never collides with an upload file. */ +export const recordingUploadPlaceholderPath = (recordingId: number, uploadName: string): string => + join(recordingUploadDir(recordingId), PLACEHOLDER_DIR, uploadName) + +/** + * Fail fast on a disk that cannot hold the declared file plus margin. The + * measurement is an optimization, not a gate: a `statfs` failure (odd + * filesystem) skips it and lets the write path hit the real bound. + */ +export const assertRecordingUploadDiskRoom = async (expectedBytes: number): Promise => { + let freeBytes: number + try { + const stats = await statfs(tmpdir()) + freeBytes = Number(stats.bavail) * Number(stats.bsize) + } catch { + return + } + if (!recordingUploadFitsInDisk({ freeBytes, expectedBytes })) { + throw new Error(RECORDING_UPLOAD_NO_SPACE_MESSAGE) + } +} + +/** Creates the empty appended file the chunks will grow and persists the state. */ +export const createRecordingUploadSession = async ( + recordingId: number, + session: RecordingUploadSession, +): Promise => { + const dir = recordingUploadDir(recordingId) + await mkdir(dir, { recursive: true }) + await writeFile(join(dir, SESSION_STATE_FILE), JSON.stringify(session)) + await writeFile(recordingUploadFilePath(recordingId, session.uploadName), '') +} + +/** Null when the session state is missing or unreadable (lost/never started). */ +export const readRecordingUploadSession = async ( + recordingId: number, +): Promise => { + try { + const raw = await readFile(join(recordingUploadDir(recordingId), SESSION_STATE_FILE), 'utf8') + const parsed = JSON.parse(raw) as Partial + if ( + typeof parsed.uploadName !== 'string' || + parsed.uploadName.length === 0 || + typeof parsed.expectedBytes !== 'number' || + !Number.isFinite(parsed.expectedBytes) || + parsed.expectedBytes < 0 + ) { + return null + } + return { uploadName: parsed.uploadName, expectedBytes: parsed.expectedBytes } + } catch { + return null + } +} + +/** Best-effort removal of the whole session (finalize, abort and reaper paths). */ +export const removeRecordingUploadSession = async (recordingId: number): Promise => { + await rm(recordingUploadDir(recordingId), { recursive: true, force: true }).catch(() => undefined) +} diff --git a/tests/e2e/campaignSpeechAcervo.e2e.spec.ts b/tests/e2e/campaignSpeechAcervo.e2e.spec.ts index 6e141f94..8169741d 100644 --- a/tests/e2e/campaignSpeechAcervo.e2e.spec.ts +++ b/tests/e2e/campaignSpeechAcervo.e2e.spec.ts @@ -748,23 +748,30 @@ test.describe('communication vertical (C154/C162)', () => { expect(denied.status()).toBe(404) }) - test('the upload route refuses cross-origin and advisor actors', async ({ + test('the upload routes refuse cross-origin and advisor actors', async ({ campaign, campaignRequest, }) => { const user = await campaign.fixtures.createCampaignUser('communicator') const request = await campaignRequest(user, user.password) const uploadUrl = '/campanha/comunicacao/acervo/gravacoes/enviar?title=Teste&filename=t.mp4' + const chunkUrl = '/campanha/comunicacao/acervo/gravacoes/enviar/1?index=0' const crossOrigin = await request.post(uploadUrl, { headers: { Origin: 'https://evil.example' }, }) expect(crossOrigin.status()).toBe(403) + const crossOriginChunk = await request.post(chunkUrl, { + headers: { Origin: 'https://evil.example' }, + }) + expect(crossOriginChunk.status()).toBe(403) const advisor = await campaign.fixtures.createCampaignUser('advisor') const advisorRequest = await campaignRequest(advisor, advisor.password) const denied = await advisorRequest.post(uploadUrl) expect(denied.status()).toBe(403) + const deniedChunk = await advisorRequest.post(chunkUrl) + expect(deniedChunk.status()).toBe(403) }) test('the grouped detail renders the speaker blocks, the warning and the identify action', async ({ diff --git a/tests/int/recording.int.spec.ts b/tests/int/recording.int.spec.ts index fd6d6c7f..a437b1a2 100644 --- a/tests/int/recording.int.spec.ts +++ b/tests/int/recording.int.spec.ts @@ -26,6 +26,11 @@ vi.mock('@/utilities/campaignAuth', () => ({ })) import { GET } from '@/app/(campaign)/campanha/(app)/comunicacao/acervo/gravacoes/[id]/arquivo/route' +import { + DELETE as abortUploadRoute, + POST as receiveChunkRoute, +} from '@/app/(campaign)/campanha/(app)/comunicacao/acervo/gravacoes/enviar/[id]/route' +import { POST as startUploadRoute } from '@/app/(campaign)/campanha/(app)/comunicacao/acervo/gravacoes/enviar/route' import { deleteRecordingForActor, getRecordingStatusesForActor, @@ -34,8 +39,11 @@ import { } from '@/app/(campaign)/campanha/actions/recording' import { getMunicipalityCatalogEntry } from '@/lib/municipalityCatalog' import { + RECORDING_CHUNK_OUT_OF_ORDER_MESSAGE, + RECORDING_CHUNK_TOO_LARGE_MESSAGE, RECORDING_FORBIDDEN_MESSAGE, RECORDING_SPEAKER_UNKNOWN_MESSAGE, + RECORDING_UPLOAD_SIZE_MISMATCH_MESSAGE, } from '@/lib/schemas/recording' import { normalizeForSearch } from '@/lib/speechSearch' import type { CampaignUser, RecordingMedia } from '@/payload-types' @@ -51,7 +59,11 @@ import { loadRecordingFilterOptions, loadRecordingsPageData, } from '@/utilities/recordings/recordingPageData' -import { receiveRecordingUpload } from '@/utilities/recordings/recordingUpload' +import { receiveRecordingChunk, startRecordingUpload } from '@/utilities/recordings/recordingUpload' +import { + createRecordingUploadSession, + readRecordingUploadSession, +} from '@/utilities/recordings/recordingUploadSession' import { installCampaignFixtures } from '../helpers/campaignFixtures' @@ -325,29 +337,56 @@ describe('uploaded recordings (C199)', () => { expect(segments.docs).toHaveLength(0) }) - it('streams an upload to the private media and hands the row to the job', async () => { - const bytes = Buffer.from('raw-body-upload') - const body = new ReadableStream({ + const streamOf = (bytes: Buffer): ReadableStream => + new ReadableStream({ start(controller) { controller.enqueue(bytes) controller.close() }, }) + + it('receives a recording in ordered parts and hands the row to the job on the last one', async () => { + const chunks = [Buffer.from('part-'), Buffer.from('by-pa'), Buffer.from('rt')] + const bytes = Buffer.concat(chunks) const scheduled: number[] = [] - const { id } = await receiveRecordingUpload({ + const { id } = await startRecordingUpload({ payload, actor: communicator, metadata: { title: 'Giro pelo interior', filename: 'giro interior.m4v', recordedAt: undefined, + size: bytes.length, }, - body, - startJob: (recordingId) => scheduled.push(recordingId), }) createdRecordingIds.add(id) + // The row is visible as `uploading` right after the start, before any part. + const started = await payload.findByID({ + collection: 'recording', + id, + depth: 0, + overrideAccess: true, + }) + expect(started.status).toBe('uploading') + expect(started.title).toBe('Giro pelo interior') + + const receive = (index: number, body: Buffer) => + receiveRecordingChunk({ + payload, + actor: communicator, + recordingId: id, + index, + body: streamOf(body), + chunkSize: chunks[0].length, + startJob: (recordingId) => scheduled.push(recordingId), + }) + + await expect(receive(0, chunks[0])).resolves.toEqual({ done: false }) + await expect(receive(1, chunks[1])).resolves.toEqual({ done: false }) + await expect(receive(2, chunks[2])).resolves.toEqual({ done: true }) + const recording = await payload.findByID({ collection: 'recording', id, @@ -366,30 +405,253 @@ describe('uploaded recordings (C199)', () => { if (media && typeof media === 'object') { createdMediaIds.add(media.id) expect(media.filename).toContain('giro_interior') + expect(media.filesize).toBe(bytes.length) } + // The temp session is gone once the upload was committed. + await expect(readRecordingUploadSession(id)).resolves.toBeNull() + const response = await callFileRoute({ recordingId: id, user: communicator }) expect(response.status).toBe(200) expect(Buffer.from(await response.arrayBuffer())).toEqual(bytes) }) - it('accepts a recording above the payload buffer boundary through the multipart seam', async () => { - const bytes = Buffer.from('plenaria-de-horas') - const body = new ReadableStream({ - start(controller) { - controller.enqueue(bytes) - controller.close() + it('serves the chunked upload wire contract through the two routes', async () => { + const bytes = Buffer.from('rota-fatiada') + getCampaignUserMock.mockResolvedValue(communicator) + + const params = new URLSearchParams({ + title: 'Via rota', + filename: 'via rota.mp4', + size: String(bytes.length), + }) + const startResponse = await startUploadRoute( + new Request( + `http://localhost/campanha/comunicacao/acervo/gravacoes/enviar?${params.toString()}`, + { method: 'POST' }, + ), + ) + expect(startResponse.status).toBe(200) + const started = (await startResponse.json()) as { + status: string + recording: { id: number; status: string } + chunkSize: number + } + expect(started.status).toBe('success') + expect(started.recording.status).toBe('uploading') + expect(started.chunkSize).toBeGreaterThan(0) + createdRecordingIds.add(started.recording.id) + + const chunkResponse = await receiveChunkRoute( + new Request( + `http://localhost/campanha/comunicacao/acervo/gravacoes/enviar/${started.recording.id}?index=0`, + { method: 'POST', body: bytes }, + ), + { params: Promise.resolve({ id: String(started.recording.id) }) }, + ) + expect(chunkResponse.status).toBe(200) + const done = (await chunkResponse.json()) as { + status: string + done: boolean + recording: { id: number; status: string } + } + expect(done.status).toBe('success') + expect(done.done).toBe(true) + expect(done.recording.status).toBe('processing') + + const recording = await payload.findByID({ + collection: 'recording', + id: started.recording.id, + depth: 1, + overrideAccess: true, + }) + if (recording.media && typeof recording.media === 'object') { + createdMediaIds.add(recording.media.id) + } + }) + + it('aborts an in-flight upload and never touches a finalized recording', async () => { + const bytes = Buffer.from('abortar') + getCampaignUserMock.mockResolvedValue(communicator) + + const aborted = await startRecordingUpload({ + payload, + actor: communicator, + metadata: { + title: 'Abortada', + filename: 'abortada.mp4', + recordedAt: undefined, + size: bytes.length, }, }) + createdRecordingIds.add(aborted.id) + + const abortResponse = await abortUploadRoute( + new Request(`http://localhost/campanha/comunicacao/acervo/gravacoes/enviar/${aborted.id}`, { + method: 'DELETE', + }), + { params: Promise.resolve({ id: String(aborted.id) }) }, + ) + expect(abortResponse.status).toBe(200) + expect(await abortResponse.json()).toEqual({ status: 'success', aborted: true }) + + const rows = await payload.find({ + collection: 'recording', + where: { id: { equals: aborted.id } }, + overrideAccess: true, + }) + expect(rows.docs).toHaveLength(0) + await expect(readRecordingUploadSession(aborted.id)).resolves.toBeNull() + + // A finalized recording survives a stray abort (e.g. the dialog losing the + // response of the part that completed the upload). + const kept = await startRecordingUpload({ + payload, + actor: communicator, + metadata: { + title: 'Mantida', + filename: 'mantida.mp4', + recordedAt: undefined, + size: bytes.length, + }, + }) + createdRecordingIds.add(kept.id) + await receiveRecordingChunk({ + payload, + actor: communicator, + recordingId: kept.id, + index: 0, + body: streamOf(bytes), + chunkSize: bytes.length, + startJob: () => undefined, + }) + + const stray = await abortUploadRoute( + new Request(`http://localhost/campanha/comunicacao/acervo/gravacoes/enviar/${kept.id}`, { + method: 'DELETE', + }), + { params: Promise.resolve({ id: String(kept.id) }) }, + ) + expect(stray.status).toBe(200) + const keptRow = await payload.findByID({ + collection: 'recording', + id: kept.id, + depth: 1, + overrideAccess: true, + }) + expect(keptRow.status).toBe('processing') + if (keptRow.media && typeof keptRow.media === 'object') { + createdMediaIds.add(keptRow.media.id) + } + }) + + it('refuses a part out of order and discards the session', async () => { + const { id } = await startRecordingUpload({ + payload, + actor: communicator, + metadata: { title: 'Fora de ordem', filename: 'ordem.mp4', recordedAt: undefined, size: 10 }, + }) + createdRecordingIds.add(id) + + await expect( + receiveRecordingChunk({ + payload, + actor: communicator, + recordingId: id, + index: 3, + body: streamOf(Buffer.from('xxxxx')), + chunkSize: 5, + }), + ).rejects.toThrow(RECORDING_CHUNK_OUT_OF_ORDER_MESSAGE) + + // A refused part aborts the whole session: no phantom "Enviando" row and + // no temp file left behind. + const rows = await payload.find({ + collection: 'recording', + where: { id: { equals: id } }, + overrideAccess: true, + }) + expect(rows.docs).toHaveLength(0) + await expect(readRecordingUploadSession(id)).resolves.toBeNull() + }) + + it('refuses a part bigger than the agreed part size or beyond the declared size', async () => { + const { id } = await startRecordingUpload({ + payload, + actor: communicator, + metadata: { title: 'Além do teto', filename: 'alem.mp4', recordedAt: undefined, size: 8 }, + }) + createdRecordingIds.add(id) + + await expect( + receiveRecordingChunk({ + payload, + actor: communicator, + recordingId: id, + index: 0, + body: streamOf(Buffer.alloc(6)), + chunkSize: 5, + }), + ).rejects.toThrow(RECORDING_CHUNK_TOO_LARGE_MESSAGE) + + const restarted = await startRecordingUpload({ + payload, + actor: communicator, + metadata: { + title: 'Além do declarado', + filename: 'declarado.mp4', + recordedAt: undefined, + size: 8, + }, + }) + createdRecordingIds.add(restarted.id) + + await expect( + receiveRecordingChunk({ + payload, + actor: communicator, + recordingId: restarted.id, + index: 0, + body: streamOf(Buffer.from('12345')), + chunkSize: 5, + }), + ).resolves.toEqual({ done: false }) + await expect( + receiveRecordingChunk({ + payload, + actor: communicator, + recordingId: restarted.id, + index: 1, + body: streamOf(Buffer.from('67890')), + chunkSize: 5, + }), + ).rejects.toThrow(RECORDING_UPLOAD_SIZE_MISMATCH_MESSAGE) + }) + + it('accepts a recording above the payload buffer boundary through the multipart seam', async () => { + const bytes = Buffer.from('plenaria-de-horas') const scheduled: number[] = [] const uploaded: { filename?: string; filesize: number }[] = [] const removed: string[] = [] - const { id } = await receiveRecordingUpload({ + const { id } = await startRecordingUpload({ payload, actor: communicator, - metadata: { title: 'Sessão longa', filename: 'sessao longa.MP4', recordedAt: undefined }, - body, + metadata: { + title: 'Sessão longa', + filename: 'sessao longa.MP4', + recordedAt: undefined, + size: bytes.length, + }, + }) + createdRecordingIds.add(id) + + const { done } = await receiveRecordingChunk({ + payload, + actor: communicator, + recordingId: id, + index: 0, + body: streamOf(bytes), largeThresholdBytes: 0, startJob: (recordingId) => scheduled.push(recordingId), uploadLarge: async ({ filename }) => { @@ -405,7 +667,7 @@ describe('uploaded recordings (C199)', () => { removed.push(filename) }, }) - createdRecordingIds.add(id) + expect(done).toBe(true) const recording = await payload.findByID({ collection: 'recording', @@ -431,20 +693,26 @@ describe('uploaded recordings (C199)', () => { }) it('removes the multipart object and leaves no phantom row when the large upload fails', async () => { - const body = new ReadableStream({ - start(controller) { - controller.enqueue(Buffer.from('falha-grande')) - controller.close() + const removed: string[] = [] + const { id } = await startRecordingUpload({ + payload, + actor: communicator, + metadata: { + title: 'Grande que falha', + filename: 'falha.mp4', + recordedAt: undefined, + size: 12, }, }) - const removed: string[] = [] + createdRecordingIds.add(id) await expect( - receiveRecordingUpload({ + receiveRecordingChunk({ payload, actor: communicator, - metadata: { title: 'Grande que falha', filename: 'falha.mp4', recordedAt: undefined }, - body, + recordingId: id, + index: 0, + body: streamOf(Buffer.from('falha-grande')), largeThresholdBytes: 0, startJob: () => undefined, uploadLarge: async () => { @@ -464,6 +732,7 @@ describe('uploaded recordings (C199)', () => { expect(rows.docs).toHaveLength(0) expect(removed).toHaveLength(1) expect(removed[0]).toContain('falha') + await expect(readRecordingUploadSession(id)).resolves.toBeNull() }) it('preserves a long recording transcript and its search text', async () => { @@ -797,12 +1066,18 @@ describe('uploaded recordings (C199)', () => { }), ).toBe(false) - // An `uploading` row abandoned mid-stream has no usable file: reaped away. + // An `uploading` row abandoned mid-stream has no usable file: reaped away, + // and its chunked-upload session dir goes with it. const orphan = await createRecording({ status: 'uploading', title: 'Upload interrompido', withMedia: false, }) + await createRecordingUploadSession(orphan.recording.id, { + uploadName: 'interrompido.mp4', + expectedBytes: 1024, + }) + expect(await readRecordingUploadSession(orphan.recording.id)).not.toBeNull() expect( await reapStaleRecording(payload, { id: orphan.recording.id, @@ -814,6 +1089,7 @@ describe('uploaded recordings (C199)', () => { await expect( payload.findByID({ collection: 'recording', id: orphan.recording.id, overrideAccess: true }), ).rejects.toThrow() + await expect(readRecordingUploadSession(orphan.recording.id)).resolves.toBeNull() }) it('polls the visible statuses and deletes a recording with its media', async () => { diff --git a/tests/unit/codebaseConventions.unit.spec.ts b/tests/unit/codebaseConventions.unit.spec.ts index 92001e36..23caee77 100644 --- a/tests/unit/codebaseConventions.unit.spec.ts +++ b/tests/unit/codebaseConventions.unit.spec.ts @@ -228,7 +228,11 @@ describe('campaign JSON mutation route convention', () => { ], [ 'src/app/(campaign)/campanha/(app)/comunicacao/acervo/gravacoes/enviar/route.ts', - 'raw-body recording upload (the video streams to disk, metadata in the query) — cookie-authenticated via campaign-token + explicit same-origin check; cannot ride the JSON wrapper', + 'raw-body recording upload session start (bodyless; metadata in the query) — cookie-authenticated via campaign-token + explicit same-origin check; cannot ride the JSON wrapper', + ], + [ + 'src/app/(campaign)/campanha/(app)/comunicacao/acervo/gravacoes/enviar/[id]/route.ts', + 'one raw-body part of a chunked recording upload (the part streams to disk, index in the query) + idempotent abort of the session (DELETE) — cookie-authenticated via campaign-token + explicit same-origin check; cannot ride the JSON wrapper', ], [ 'src/app/(campaign)/campanha/(app)/comunicacao/conteudos/enviar/route.ts', diff --git a/tests/unit/recording.unit.spec.ts b/tests/unit/recording.unit.spec.ts index 2bd5de98..b5e6f642 100644 --- a/tests/unit/recording.unit.spec.ts +++ b/tests/unit/recording.unit.spec.ts @@ -10,9 +10,12 @@ import { RECORDING_FAILURE_INTERRUPTED, RECORDING_STATUSES, RECORDING_STEPS, + RECORDING_UPLOAD_CHUNK_BYTES, + RECORDING_UPLOAD_EDGE_LIMIT_BYTES, recordingFailureMessage, recordingFileTypeAllowed, recordingStatusLabels, + recordingUploadFitsInDisk, toRecordingViewModel, } from '@/lib/recording' import { @@ -79,6 +82,27 @@ describe('recording vocabulary (C199)', () => { expect(formatRecordingFileSize(24 * 1024 ** 3)).toBe('24 GB') }) + it('keeps every upload request under the production edge ceiling', () => { + // Production is behind the Cloudflare Tunnel: the edge answers `413 Payload + // Too Large` for a request body above ~100 MB BEFORE the app sees it (probed + // against prod, a 150 MB body answered 413). The dialog slices the file into + // parts of `RECORDING_UPLOAD_CHUNK_BYTES`, so raising it past the edge limit + // would silently break every multi-GB recording upload again. + expect(RECORDING_UPLOAD_CHUNK_BYTES).toBeGreaterThanOrEqual(1024 * 1024) + expect(RECORDING_UPLOAD_CHUNK_BYTES).toBeLessThan(RECORDING_UPLOAD_EDGE_LIMIT_BYTES) + }) + + it('demands room for the declared file plus a margin before the upload starts', () => { + const sixGiB = 6 * 1024 ** 3 + expect(recordingUploadFitsInDisk({ freeBytes: 20 * 1024 ** 3, expectedBytes: sixGiB })).toBe( + true, + ) + expect( + recordingUploadFitsInDisk({ freeBytes: sixGiB + 100 * 1024 ** 2, expectedBytes: sixGiB }), + ).toBe(false) + expect(recordingUploadFitsInDisk({ freeBytes: Number.NaN, expectedBytes: 0 })).toBe(false) + }) + it('formats the byte range below 1 KB honestly', () => { expect(formatRecordingFileSize(0)).toBe('0 B') expect(formatRecordingFileSize(900)).toBe('900 B')