mirror of
https://github.com/coder/coder.git
synced 2026-06-03 13:08:25 +00:00
Michael Suchacz
85792d08bc
This PR adds an opinionated harness-engineering layer for agent-driven workflows: a small set of agent-readable docs, mechanical structure checks, structured CI failure summaries, an architecture-lint umbrella, and per-worktree dev-server isolation. The goal is to make local dev, tests, and CI mechanically inspectable by agents without changing app runtime behavior. ## What landed **Agent docs and navigation** - `.claude/docs/OBSERVABILITY.md`, `.claude/docs/DEV_ISOLATION.md`, `.claude/docs/AGENT_FAILURES.md`: task-oriented guides for logs, tracing, Prometheus, dev-server isolation, and a seeded failure catalog. - `AGENTS.md`: added an `Agent navigation` block, then trimmed the file from 375 to 229 lines by migrating duplicated detail into `WORKFLOWS.md`, `GO.md`, `TESTING.md`, and `DATABASE.md`. The user-managed custom-instructions block is preserved. - `.agents/docs`: symlink mirror of `.claude/docs` for agent runtimes that look under `.agents`. **Mechanical checks** - `scripts/check_agents_structure.sh`: validates `@...` references in tracked `AGENTS.md` files and warns when root grows past 600 lines. Wired as `make lint/agents` and into `make lint`. - `scripts/audit-agent-readiness.sh`: report-first audit of harness readiness. Currently `10 ok, 0 warn, 0 fail`. - `scripts/check_architecture.sh` / `make lint/architecture`: umbrella architecture-lint target. Consolidates the existing `check_enterprise_imports.sh` and `check_codersdk_imports.sh` so they run exactly once via the umbrella. Slot is open for new high-confidence rules. **Structured CI failure summaries** - `scripts/playwright-failure-summary.sh`: parses `site/test-results/results.json` and writes Markdown to `$GITHUB_STEP_SUMMARY` on failure. Wired into the `test-e2e` matrix job. - `scripts/go-test-failure-summary.sh`: parses `go test -json` line-delimited output the same way. Wired into `test-go-pg`, `test-go-pg-17`, and `test-go-race-pg` by injecting `gotestsum --jsonfile` in the workflow without touching `Makefile`. JSON also uploaded as a CI artifact on failure. - `site/e2e/playwright.config.ts`: enables `screenshot: only-on-failure`, `trace: retain-on-failure`, JSON reporter, and HTML reporter alongside existing reporters. - `.github/workflows/ci.yaml`: failure artifact uploads for Playwright now use `if: failure()` and predictable names (`playwright-artifacts-<variant>-<sha>`). **Per-worktree dev-server isolation** (`scripts/develop/main.go`) - Deterministic FNV-64a hash of the worktree path produces a port offset in `[0, 1000)` (50 buckets, step 20 to avoid API/proxy overlap across adjacent buckets). - Offset is applied only to defaults; both env vars (`CODER_DEV_PORT`, `CODER_DEV_WEB_PORT`, `CODER_DEV_PROXY_PORT`, `CODER_DEV_PROMETHEUS_PORT`) and CLI flags retain priority. - Hardcoded ports `9090` (embedded Prometheus UI) and `12345` (Delve) are unchanged by design. - Startup banner shows each port's source: `default`, `offset`, or `explicit`. - Unit tests in `scripts/develop/main_test.go` cover determinism, bounds, no-overlap across the four ports, and explicit-skip behavior. - State (`.coderv2/`) was already worktree-isolated via `os.Getwd()`, so no state-dir changes were needed. ## Validation `make lint/agents`, `make lint/architecture`, `make lint/emdash`, `bash scripts/audit-agent-readiness.sh` (10 ok, 0 warn, 0 fail), `shellcheck` on all 5 new scripts, `go test ./scripts/develop/...`, and `js-yaml` parse of `ci.yaml` all pass. Synthetic fixtures verify both failure-summary scripts handle empty/missing input (silent exit 0), ANSI-stripped output, and parent/subtest formatting. ## Known follow-ups (deferred) - Frontend Storybook/Vitest failure summary: lowest-leverage slice of the failure-summary work. Skipping until observed pain. - Architecture lint currently only delegates to existing import checks; new rules (`InTx` outer-store detection, swagger-annotation lint) plug in as needed. - 50 port-offset buckets means two worktree paths can occasionally collide. The DEV_ISOLATION doc tells users to set the relevant env var when this happens. > Mux opened this PR on Mike's behalf.
131 lines
3.2 KiB
Bash
Executable File
131 lines
3.2 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
set -euo pipefail
|
|
# shellcheck source=scripts/lib.sh
|
|
# shellcheck disable=SC1091
|
|
source "$(dirname "${BASH_SOURCE[0]}")/lib.sh"
|
|
cdroot
|
|
|
|
usage() {
|
|
cat <<'USAGE'
|
|
Usage: scripts/audit-agent-readiness.sh [--help]
|
|
|
|
Print a report-first audit of agent harness readiness. Warnings identify
|
|
aspirational checks and do not fail the script. Missing required harness docs
|
|
fail the script. Run manually with:
|
|
|
|
bash scripts/audit-agent-readiness.sh
|
|
USAGE
|
|
}
|
|
|
|
if [[ "${1:-}" == "--help" ]]; then
|
|
usage
|
|
exit 0
|
|
fi
|
|
|
|
ok_count=0
|
|
warn_count=0
|
|
fail_count=0
|
|
|
|
ok() {
|
|
printf '[ok] %s\n' "$1"
|
|
((ok_count++)) || true
|
|
}
|
|
|
|
warn() {
|
|
printf '[warn] %s\n' "$1"
|
|
((warn_count++)) || true
|
|
}
|
|
|
|
fail() {
|
|
printf '[fail] %s\n' "$1"
|
|
((fail_count++)) || true
|
|
}
|
|
|
|
contains() {
|
|
local file="$1"
|
|
local pattern="$2"
|
|
grep -qiE "$pattern" "$file"
|
|
}
|
|
|
|
echo "Agent harness readiness audit"
|
|
echo
|
|
echo "Required harness docs"
|
|
|
|
for doc in \
|
|
".claude/docs/OBSERVABILITY.md" \
|
|
".claude/docs/DEV_ISOLATION.md" \
|
|
".claude/docs/AGENT_FAILURES.md"; do
|
|
if [[ -f "$doc" ]]; then
|
|
ok "$doc exists."
|
|
else
|
|
fail "$doc is missing."
|
|
fi
|
|
done
|
|
|
|
if [[ -L ".agents/docs" ]]; then
|
|
agents_docs_target="$(readlink ".agents/docs")"
|
|
if [[ "$agents_docs_target" == "../.claude/docs" ]]; then
|
|
ok ".agents/docs points to .claude/docs."
|
|
else
|
|
fail ".agents/docs points to $agents_docs_target, expected ../.claude/docs."
|
|
fi
|
|
else
|
|
fail ".agents/docs compatibility symlink is missing."
|
|
fi
|
|
|
|
echo
|
|
echo "Navigation and report-first checks"
|
|
|
|
if contains AGENTS.md '^##[[:space:]].*(Agent navigation|Where to look)' ||
|
|
{ grep -qF ".claude/docs/OBSERVABILITY.md" AGENTS.md &&
|
|
grep -qF ".claude/docs/DEV_ISOLATION.md" AGENTS.md &&
|
|
grep -qF ".claude/docs/AGENT_FAILURES.md" AGENTS.md; }; then
|
|
ok "Root AGENTS.md appears to include agent navigation."
|
|
else
|
|
warn "Root AGENTS.md may be missing agent navigation."
|
|
fi
|
|
|
|
if contains site/e2e/playwright.config.ts 'screenshot' &&
|
|
contains site/e2e/playwright.config.ts 'video' &&
|
|
contains site/e2e/playwright.config.ts 'trace' &&
|
|
contains site/e2e/playwright.config.ts 'failure'; then
|
|
ok "Playwright failure artifact settings appear configured."
|
|
else
|
|
warn "Playwright failure artifact settings were not all detected."
|
|
fi
|
|
|
|
if grep -qi "playwright" .github/workflows/ci.yaml &&
|
|
grep -q "upload-artifact" .github/workflows/ci.yaml &&
|
|
grep -qF "failure()" .github/workflows/ci.yaml; then
|
|
ok "E2E CI failure artifact upload appears configured."
|
|
else
|
|
warn "E2E CI failure artifact upload was not detected."
|
|
fi
|
|
|
|
if contains .claude/docs/OBSERVABILITY.md 'Prometheus' &&
|
|
contains .claude/docs/OBSERVABILITY.md 'log'; then
|
|
ok "Observability doc mentions logs and Prometheus."
|
|
else
|
|
warn "Observability doc may be missing logs or Prometheus coverage."
|
|
fi
|
|
|
|
if contains .claude/docs/DEV_ISOLATION.md 'port' &&
|
|
contains .claude/docs/DEV_ISOLATION.md 'CODER_DEV|override'; then
|
|
ok "Development isolation doc mentions ports and overrides."
|
|
else
|
|
warn "Development isolation doc may be missing ports or override coverage."
|
|
fi
|
|
|
|
if grep -q 'lint/architecture' Makefile; then
|
|
ok "Architecture lint target exists."
|
|
else
|
|
warn "Architecture lint target is not present yet."
|
|
fi
|
|
|
|
echo
|
|
printf 'Summary: %d ok, %d warn, %d fail.\n' "$ok_count" "$warn_count" "$fail_count"
|
|
|
|
if ((fail_count > 0)); then
|
|
exit 1
|
|
fi
|