From 1ebae9142873f04cb1d08176baacb4de70f80b42 Mon Sep 17 00:00:00 2001 From: Francisco Solla Date: Thu, 17 Sep 2026 01:48:29 -0300 Subject: [PATCH] fix(OPS126): auto-unblock honra o proxy do homeserver e resolve o opencode fora do PATH MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Duas causas independentes mantinham o agente de desbloqueio (OPS106) sem entregar: 1. `fetch failed` no launcher: o homeserver só alcança o GitHub pelo proxy CONNECT local (github-tunnel + http-proxy-socks) e o `fetch` global do Node ignora HTTP(S)_PROXY sem `NODE_USE_ENV_PROXY` (Node >= 24). O launcher morria lendo os jobs do run (run 35171293357) e nenhum token era criado. `scripts/auto-unblock.sh` passa a exportar `NODE_USE_ENV_PROXY=1` quando há proxy no env; o agente destacado herda. 2. exit 127 do agente: o serviço do runner não carrega o profile interativo, então `~/.opencode/bin` fica fora do PATH e o `timeout ... opencode` morria com *No such file or directory* (log unblock-35101121310.log), deixando o token `blocked` sem PR. Novo `resolveOpenCodeBinary` (OPENCODE_BIN → argv0 com path → `$HOME/.opencode/bin` → PATH) usado pelo wrapper, pelo `--check` e por um gate fail-closed no launcher (token blocked, sem provisionar worktree). Runbook + changelog OPS126; unit tests do resolver. Token órfão #1089 fechado no replay. Flakiness de e2e full segue na #1124. --- docs/changelog/2026-09-17-ops126.md | 1 + docs/ops/teqo-1313-deploy.md | 9 +++++ scripts/auto-unblock-agent.mjs | 35 ++++++++++++++++- scripts/auto-unblock.mjs | 53 ++++++++++++++++++++++++- scripts/auto-unblock.sh | 11 ++++++ scripts/lib/auto-unblock.mjs | 32 +++++++++++++++ tests/unit/autoUnblock.unit.spec.ts | 60 +++++++++++++++++++++++++++++ 7 files changed, 198 insertions(+), 3 deletions(-) create mode 100644 docs/changelog/2026-09-17-ops126.md diff --git a/docs/changelog/2026-09-17-ops126.md b/docs/changelog/2026-09-17-ops126.md new file mode 100644 index 00000000..2a55f662 --- /dev/null +++ b/docs/changelog/2026-09-17-ops126.md @@ -0,0 +1 @@ +**OPS126 — auto-unblock volta a despachar de verdade no homeserver: fetch honra o proxy e o opencode é resolvido fora do PATH (2026-09-17):** duas causas independentes mantinham o agente de desbloqueio (OPS106) sem entregar nada. **(1) `fetch failed` no launcher:** o homeserver só alcança o GitHub pelo proxy CONNECT local (`github-tunnel` + `http-proxy-socks`, `~/actions-runner/.env` com `HTTPS_PROXY/HTTP_PROXY=127.0.0.1:1081`); o `fetch` global do Node ignora `HTTP(S)_PROXY` a menos que `NODE_USE_ENV_PROXY` seja definido **antes** do processo subir (Node ≥24), então a leitura dos jobs do run caía em conexão direta e falhava de forma intermitente (run `35171293357`: 4/4 `fetch failed` e nenhum token criado). `scripts/auto-unblock.sh` passa a exportar `NODE_USE_ENV_PROXY=1` quando há proxy configurado — o agente destacado herda. **(2) exit 127 do agente:** o serviço do runner não carrega o profile interativo, então `~/.opencode/bin` não está no PATH e o `timeout ... opencode` morria com _No such file or directory_ (log `unblock-35101121310.log`), deixando o token `blocked` sem PR. Novo helper puro `resolveOpenCodeBinary` (`OPENCODE_BIN` → argv0 com path → `$HOME/.opencode/bin/` → PATH) é usado pelo wrapper e pelo `--check` (que agora acusa o binário ausente em vez de passar), e o launcher falha fechado **antes** de provisionar o worktree quando o binário não existe. Sem schema, sem UI, sem migration. Verificação: `pnpm vitest run tests/unit/autoUnblock.unit.spec.ts tests/unit/autoUnblockWorkflow.unit.spec.ts`, `bash -n scripts/auto-unblock.sh`; smoke no homeserver (`NODE_USE_ENV_PROXY=1 node -e fetch` → 200 estável). Bootstrap OPS106 §1 concluído no homeserver em 2026-09-17: opencode 1.18.31 instalado em `~/.opencode/bin` (mesmo binário da workstation, via `--binary`) e `~/.local/share/opencode/auth.json` (chmod 600) replicado — `opencode run --model deepseek/deepseek-flash` responde pelo proxy. Antes disso, qualquer token nascia `blocked`. Token `auto-unblock` #1089 (órfão `blocked` de run superado) fechado no replay. Runs vermelhos de referência: `35170301963`, `35166706444` (flakiness de e2e segue na Issue #1124). diff --git a/docs/ops/teqo-1313-deploy.md b/docs/ops/teqo-1313-deploy.md index 2875e879..a44a2cea 100644 --- a/docs/ops/teqo-1313-deploy.md +++ b/docs/ops/teqo-1313-deploy.md @@ -296,6 +296,8 @@ a correção se re-mergeia e o próximo deploy completo publica. | `deploy-staging` falha cedo (`teqo-staging.env`/serviço ausente) | Bootstrap do staging incompleto (DB, env file, serviços no compose ou environment `staging`) | Rodar o bootstrap da seção Staging; produção nunca é tocada (staging vermelho fail-closa o job `deploy-production`) | | `verify` vermelho | Regressão real na suíte full (ou flake — #882/#906) | O `auto-unblock.yml` dispara o agente sozinho (OPS106): acompanhe a Issue `auto-unblock` (token) e o PR `fix/*`. Sem PR, o token fica `blocked` — inspecione o log em `~/teqo-unblock/logs/`. Correção manual segue possível (re-merge ou `workflow_dispatch`) | | Agente de desbloqueio não disparou (token inexistente) | Bootstrap do homeserver incompleto — opencode sem auth, Node/pnpm fora do PATH do runner, PAT ausente/sem push, Docker/porta 5432, label ausente | Rodar `pnpm unblock:check` no homeserver (com `GITHUB_TOKEN` e `AUTOMERGE_PAT` exportados) e seguir a seção Auto-unblock (OPS106) abaixo | +| Launcher `auto-unblock` morre em `fetch failed` (nenhum token criado) | Node fetch ignora HTTP(S)\_PROXY sem NODE_USE_ENV_PROXY (Node ≥24); no homeserver o GitHub só sai pelo proxy CONNECT do runner | OPS126: `scripts/auto-unblock.sh` liga o `NODE_USE_ENV_PROXY` quando há proxy no env; manter `HTTPS_PROXY/HTTP_PROXY` no `~/actions-runner/.env`. Re-dispatch ou aguarde a próxima falha do `verify` | +| Agente morre com exit 127 (`opencode`: No such file or directory) | Serviço do runner não carrega o profile → `~/.opencode/bin` fora do PATH; ou opencode nem instalado | OPS126: `resolveOpenCodeBinary` (`OPENCODE_BIN` → `$HOME/.opencode/bin` → PATH); o launcher falha fechado (token `blocked`) se faltar. Instale/autentique via `pnpm unblock:check` | | Token `auto-unblock` preso (agente morto, sem PR) | Wrapper morreu (OOM/timeout/erro) — lock livre, token < TTL se torna `skip_comment` na próxima falha | O TTL de 3h reclaima o token automaticamente (comenta + fecha + novo agente). Antes disso, inspecione o log e a branch `fix/*`; para liberar na hora, feche a Issue token com um comentário | ## Auto-unblock (OPS106) @@ -321,6 +323,13 @@ o deploy não tem auto-retry. 1. **opencode no homeserver:** instalado e autenticado no usuário do runner (`opencode auth login`; o provider do modelo default — `deepseek/deepseek-flash` via env `OPENCODE_WORKTREE_MODEL` para override — precisa estar presente). + O serviço do runner **não** carrega o profile interativo, então + `~/.opencode/bin` fica fora do PATH: o OPS126 resolve o binário + (`OPENCODE_BIN` → `$HOME/.opencode/bin/opencode` → PATH) e o launcher falha + fechado se não achar. Rede: o homeserver só alcança o GitHub pelo proxy + CONNECT local (`github-tunnel` + `http-proxy-socks`; `~/actions-runner/.env` + com `HTTPS_PROXY/HTTP_PROXY`) — o `scripts/auto-unblock.sh` liga o + `NODE_USE_ENV_PROXY` para o `fetch` do Node respeitá-lo. 2. **PAT:** secret `AUTOMERGE_PAT` no repo com os escopos `contents: write`, `pull requests: write` e `issues: write` (o agente faz push/PR e o outcome comenta/fecha a Issue; o built-in token do job não sobrevive ao processo diff --git a/scripts/auto-unblock-agent.mjs b/scripts/auto-unblock-agent.mjs index 15d7dc72..1db9994c 100644 --- a/scripts/auto-unblock-agent.mjs +++ b/scripts/auto-unblock-agent.mjs @@ -14,7 +14,8 @@ */ import { spawnSync } from 'node:child_process' -import { existsSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs' +import { accessSync, constants, existsSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs' +import { homedir } from 'node:os' import { dirname, join, resolve } from 'node:path' import { fileURLToPath } from 'node:url' @@ -25,6 +26,7 @@ import { evaluateDatabaseTargets, outcomeComment, parseHeadlessDirective, + resolveOpenCodeBinary, UNBLOCK_BLOCKED_LABEL, } from './lib/auto-unblock.mjs' import { dieWithLabel, parseEqualsFlags } from './lib/cli.mjs' @@ -40,6 +42,16 @@ const unblockHome = process.env.TEQO_UNBLOCK_HOME ?? join(process.env.HOME ?? '' const say = (message) => console.log(`[auto-unblock:agent] ${message}`) +/** X_OK probe the resolver uses — a missing binary must fail closed, not 127. */ +const isExecutable = (path) => { + try { + accessSync(path, constants.X_OK) + return true + } catch { + return false + } +} + const { flags } = parseEqualsFlags(process.argv.slice(2)) const tokenIssue = Number(flags.token) const statePath = typeof flags.state === 'string' ? flags.state : null @@ -151,7 +163,26 @@ const main = async () => { childEnv.GIT_CONFIG_VALUE_0 = '!f() { echo username=x-access-token; echo password=$GITHUB_TOKEN; }; f' - const [command, ...args] = directive.argv + const [rawCommand, ...args] = directive.argv + const resolvedCommand = resolveOpenCodeBinary({ + argv0: rawCommand, + env: childEnv, + home: process.env.HOME ?? homedir(), + exists: isExecutable, + }) + if (rawCommand === 'opencode' && !resolvedCommand) { + await blockToken( + [ + 'opencode não encontrado: fora do PATH do runner e sem `$HOME/.opencode/bin/opencode`.', + '', + 'Bootstrap pendente no homeserver (instalar/autenticar opencode no usuário do runner) —', + 'rode `pnpm unblock:check` com `GITHUB_TOKEN`/`AUTOMERGE_PAT` exportados.', + ].join('\n'), + ) + die('opencode não encontrado — token blocked (bootstrap pendente).') + } + const command = resolvedCommand ?? rawCommand + say(`opencode resolver: '${rawCommand}' → ${command}`) const agentRun = runTimed( 'opencode run --command bug-fix', AGENT_TIMEOUT_SECONDS, diff --git a/scripts/auto-unblock.mjs b/scripts/auto-unblock.mjs index 7ad54837..a6be6aec 100644 --- a/scripts/auto-unblock.mjs +++ b/scripts/auto-unblock.mjs @@ -35,6 +35,7 @@ import { fileURLToPath } from 'node:url' import { classifyDeployFailure, decideUnblock, + resolveOpenCodeBinary, tokenBody, tokenTitle, UNBLOCK_BLOCKED_LABEL, @@ -75,6 +76,16 @@ const ensureLabel = async (api) => { const run = (command, args) => spawnSync(command, args, { encoding: 'utf8', timeout: 30_000 }) +/** X_OK probe for the opencode resolver (mirrors the detached wrapper). */ +const isExecutable = (path) => { + try { + accessSync(path, constants.X_OK) + return true + } catch { + return false + } +} + const port5432State = async () => { // Reuses the db-start owner's parser and container name; a Docker daemon // that is down fails here instead of silently reading as "port free". @@ -161,7 +172,25 @@ const runCheck = async () => { } return 'node_modules ok' }) - await check('opencode', version('opencode')) + await check('opencode', () => { + // The runner service PATH lacks `~/.opencode/bin`; resolve explicitly so + // this smoke is truthful (live finding: exit 127, launcher passed anyway). + const binary = resolveOpenCodeBinary({ + argv0: 'opencode', + env: process.env, + home: homedir(), + exists: isExecutable, + }) + if (!binary) { + throw new Error( + 'binário não encontrado no PATH nem em ~/.opencode/bin — instale o opencode no usuário do runner', + ) + } + const result = run(binary, ['--version']) + if (result.error) throw result.error + if (result.status !== 0) throw new Error(`${binary} saiu com ${result.status}`) + return `${(result.stdout || result.stderr || '').trim().split('\n')[0]} (${binary})` + }) await check('opencode auth', () => { const dataHome = process.env.XDG_DATA_HOME ?? join(homedir(), '.local', 'share') const authFile = join(dataHome, 'opencode', 'auth.json') @@ -297,6 +326,28 @@ const main = async () => { die('AUTOMERGE_PAT ausente — agente não disparado (token marcado blocked).') } + // Fail fast on a missing opencode BEFORE provisioning a worktree/DBs: the + // detached agent would otherwise die with exit 127 (live finding on the + // homeserver, where the runner PATH lacks `~/.opencode/bin`). + const opencodeBinary = resolveOpenCodeBinary({ + argv0: 'opencode', + env: process.env, + home: homedir(), + exists: isExecutable, + }) + if (!opencodeBinary) { + await api.addComment( + created.number, + [ + 'opencode não encontrado no runner (fora do PATH e sem `$HOME/.opencode/bin/opencode`) —', + 'bootstrap pendente no homeserver. Instale/autentique o opencode no usuário do runner e rode', + '`pnpm unblock:check`. Fail-closed: nenhum worktree foi provisionado.', + ].join('\n'), + ) + await api.addLabels(created.number, [UNBLOCK_BLOCKED_LABEL]) + die('opencode ausente — agente não disparado (token marcado blocked).') + } + const statePath = join(stateDir, `unblock-${runId}.json`) writeFileSync( statePath, diff --git a/scripts/auto-unblock.sh b/scripts/auto-unblock.sh index dfb0e008..eefebbb4 100755 --- a/scripts/auto-unblock.sh +++ b/scripts/auto-unblock.sh @@ -22,4 +22,15 @@ flock -n 9 || LOCK_BUSY=1 # Hand the exact lock path to the orchestrator (its lock probe uses it). export TEQO_UNBLOCK_LOCK="$LOCK_FILE" + +# Node's global fetch ignores HTTP(S)_PROXY unless NODE_USE_ENV_PROXY is set +# BEFORE the process starts (Node >= 24). The homeserver reaches GitHub only +# through the runner's local CONNECT proxy (github-tunnel + http-proxy-socks), +# and a direct fetch fails intermittently (live finding: launcher run +# 35171293357 died on `fetch failed` reading the run's jobs). Enable the env +# proxy whenever the runner configured one; the detached agent inherits it. +if [[ -n "${HTTPS_PROXY:-}${HTTP_PROXY:-}" ]]; then + export NODE_USE_ENV_PROXY="${NODE_USE_ENV_PROXY:-1}" +fi + exec node scripts/auto-unblock.mjs "--lock-busy=$LOCK_BUSY" diff --git a/scripts/lib/auto-unblock.mjs b/scripts/lib/auto-unblock.mjs index 549063d2..68c09824 100644 --- a/scripts/lib/auto-unblock.mjs +++ b/scripts/lib/auto-unblock.mjs @@ -6,6 +6,8 @@ * guardrails without a GitHub token, Docker or opencode. */ +import { join } from 'node:path' + import { databaseHostname, isLocalDatabaseUrl, TEST_DATABASE_NAME_RE } from './cli.mjs' import { isGeneratedDatabaseName } from './worktree-env.mjs' @@ -22,6 +24,36 @@ export const UNBLOCK_BLOCKED_LABEL = 'blocked' */ export const UNBLOCK_TOKEN_TTL_MS = 3 * 60 * 60 * 1000 +/** + * Locate the `opencode` executable for headless launches. The self-hosted + * runner service does not source the interactive shell profile, so + * `$HOME/.opencode/bin` is absent from its PATH — the OPS106 agent died with + * exit 127 (`timeout: failed to run command 'opencode'`) on the homeserver + * even though opencode was installed for the runner user + * (`unblock-35101121310.log`). Resolution order: explicit `OPENCODE_BIN`, a + * path-like argv0, the default install dir under HOME, then the PATH entries. + * Returns null when nothing matches — the caller fails closed instead of + * spawning a doomed `timeout ... opencode`. + * + * @param {{ argv0?: string, env?: Record, home?: string, exists?: (path: string) => boolean }} [options] + * @returns {string | null} + */ +export const resolveOpenCodeBinary = ({ + argv0 = 'opencode', + env = {}, + home = '', + exists = () => false, +} = {}) => { + if (env.OPENCODE_BIN) return env.OPENCODE_BIN + if (argv0.includes('/')) return argv0 + const candidates = [] + if (home) candidates.push(join(home, '.opencode', 'bin', argv0)) + for (const dir of String(env.PATH ?? '').split(':')) { + if (dir) candidates.push(join(dir, argv0)) + } + return candidates.find((candidate) => exists(candidate)) ?? null +} + /** * Local ports that are only ever the homeserver's stack DB proxies — defense * in depth on top of the name check; the owner of the mapping is diff --git a/tests/unit/autoUnblock.unit.spec.ts b/tests/unit/autoUnblock.unit.spec.ts index 07b6498f..4b432e6e 100644 --- a/tests/unit/autoUnblock.unit.spec.ts +++ b/tests/unit/autoUnblock.unit.spec.ts @@ -11,6 +11,7 @@ import { evaluateDatabaseTargets, outcomeComment, parseHeadlessDirective, + resolveOpenCodeBinary, tokenBody, tokenTitle, UNBLOCK_LABEL, @@ -279,6 +280,65 @@ describe('classifyOutcome', () => { }) }) +describe('resolveOpenCodeBinary (o PATH do runner não tem ~/.opencode/bin)', () => { + const home = '/home/runner' + const binDir = `${home}/.opencode/bin/opencode` + + it('OPENCODE_BIN tem precedência', () => { + expect( + resolveOpenCodeBinary({ + env: { OPENCODE_BIN: '/custom/opencode', PATH: '/usr/bin' }, + home, + exists: () => true, + }), + ).toBe('/custom/opencode') + }) + + it('argv0 com path é devolvido como veio (o runner pode usar caminho absoluto)', () => { + expect( + resolveOpenCodeBinary({ + argv0: '/opt/opencode/bin/opencode', + env: {}, + home: '', + exists: () => false, + }), + ).toBe('/opt/opencode/bin/opencode') + }) + + it('cai no install padrão ~/.opencode/bin quando fora do PATH (causa do exit 127)', () => { + expect( + resolveOpenCodeBinary({ + argv0: 'opencode', + env: { PATH: '/usr/bin:/bin' }, + home, + exists: (candidate) => candidate === binDir, + }), + ).toBe(binDir) + }) + + it('encontra o opencode no PATH quando presente', () => { + expect( + resolveOpenCodeBinary({ + argv0: 'opencode', + env: { PATH: '/usr/bin:/usr/local/bin' }, + home, + exists: (candidate) => candidate === '/usr/local/bin/opencode', + }), + ).toBe('/usr/local/bin/opencode') + }) + + it('sem candidato existente → null (caller falha fechado, sem `timeout ... opencode`)', () => { + expect( + resolveOpenCodeBinary({ + argv0: 'opencode', + env: { PATH: '/usr/bin' }, + home, + exists: () => false, + }), + ).toBeNull() + }) +}) + describe('parseHeadlessDirective', () => { const valid = JSON.stringify({ dir: '/home/x/teqo-unblock/worktrees/fix/abc',