mirror of
https://github.com/coder/coder.git
synced 2026-06-03 04:58:23 +00:00
85792d08bc
This PR adds an opinionated harness-engineering layer for agent-driven workflows: a small set of agent-readable docs, mechanical structure checks, structured CI failure summaries, an architecture-lint umbrella, and per-worktree dev-server isolation. The goal is to make local dev, tests, and CI mechanically inspectable by agents without changing app runtime behavior. ## What landed **Agent docs and navigation** - `.claude/docs/OBSERVABILITY.md`, `.claude/docs/DEV_ISOLATION.md`, `.claude/docs/AGENT_FAILURES.md`: task-oriented guides for logs, tracing, Prometheus, dev-server isolation, and a seeded failure catalog. - `AGENTS.md`: added an `Agent navigation` block, then trimmed the file from 375 to 229 lines by migrating duplicated detail into `WORKFLOWS.md`, `GO.md`, `TESTING.md`, and `DATABASE.md`. The user-managed custom-instructions block is preserved. - `.agents/docs`: symlink mirror of `.claude/docs` for agent runtimes that look under `.agents`. **Mechanical checks** - `scripts/check_agents_structure.sh`: validates `@...` references in tracked `AGENTS.md` files and warns when root grows past 600 lines. Wired as `make lint/agents` and into `make lint`. - `scripts/audit-agent-readiness.sh`: report-first audit of harness readiness. Currently `10 ok, 0 warn, 0 fail`. - `scripts/check_architecture.sh` / `make lint/architecture`: umbrella architecture-lint target. Consolidates the existing `check_enterprise_imports.sh` and `check_codersdk_imports.sh` so they run exactly once via the umbrella. Slot is open for new high-confidence rules. **Structured CI failure summaries** - `scripts/playwright-failure-summary.sh`: parses `site/test-results/results.json` and writes Markdown to `$GITHUB_STEP_SUMMARY` on failure. Wired into the `test-e2e` matrix job. - `scripts/go-test-failure-summary.sh`: parses `go test -json` line-delimited output the same way. Wired into `test-go-pg`, `test-go-pg-17`, and `test-go-race-pg` by injecting `gotestsum --jsonfile` in the workflow without touching `Makefile`. JSON also uploaded as a CI artifact on failure. - `site/e2e/playwright.config.ts`: enables `screenshot: only-on-failure`, `trace: retain-on-failure`, JSON reporter, and HTML reporter alongside existing reporters. - `.github/workflows/ci.yaml`: failure artifact uploads for Playwright now use `if: failure()` and predictable names (`playwright-artifacts-<variant>-<sha>`). **Per-worktree dev-server isolation** (`scripts/develop/main.go`) - Deterministic FNV-64a hash of the worktree path produces a port offset in `[0, 1000)` (50 buckets, step 20 to avoid API/proxy overlap across adjacent buckets). - Offset is applied only to defaults; both env vars (`CODER_DEV_PORT`, `CODER_DEV_WEB_PORT`, `CODER_DEV_PROXY_PORT`, `CODER_DEV_PROMETHEUS_PORT`) and CLI flags retain priority. - Hardcoded ports `9090` (embedded Prometheus UI) and `12345` (Delve) are unchanged by design. - Startup banner shows each port's source: `default`, `offset`, or `explicit`. - Unit tests in `scripts/develop/main_test.go` cover determinism, bounds, no-overlap across the four ports, and explicit-skip behavior. - State (`.coderv2/`) was already worktree-isolated via `os.Getwd()`, so no state-dir changes were needed. ## Validation `make lint/agents`, `make lint/architecture`, `make lint/emdash`, `bash scripts/audit-agent-readiness.sh` (10 ok, 0 warn, 0 fail), `shellcheck` on all 5 new scripts, `go test ./scripts/develop/...`, and `js-yaml` parse of `ci.yaml` all pass. Synthetic fixtures verify both failure-summary scripts handle empty/missing input (silent exit 0), ANSI-stripped output, and parent/subtest formatting. ## Known follow-ups (deferred) - Frontend Storybook/Vitest failure summary: lowest-leverage slice of the failure-summary work. Skipping until observed pain. - Architecture lint currently only delegates to existing import checks; new rules (`InTx` outer-store detection, swagger-annotation lint) plug in as needed. - 50 port-offset buckets means two worktree paths can occasionally collide. The DEV_ISOLATION doc tells users to set the relevant env var when this happens. > Mux opened this PR on Mike's behalf.
97 lines
2.2 KiB
Bash
Executable File
97 lines
2.2 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
set -euo pipefail
|
|
# shellcheck source=scripts/lib.sh
|
|
# shellcheck disable=SC1091
|
|
source "$(dirname "${BASH_SOURCE[0]}")/lib.sh"
|
|
cdroot
|
|
|
|
echo "--- check agent docs structure"
|
|
|
|
required_docs=(
|
|
".claude/docs/OBSERVABILITY.md"
|
|
".claude/docs/DEV_ISOLATION.md"
|
|
".claude/docs/AGENT_FAILURES.md"
|
|
)
|
|
|
|
fail=0
|
|
|
|
for doc in "${required_docs[@]}"; do
|
|
if [[ ! -f "$doc" ]]; then
|
|
echo "error: required harness doc is missing: $doc"
|
|
fail=1
|
|
fi
|
|
done
|
|
|
|
if [[ ! -L ".agents/docs" ]]; then
|
|
echo "error: agent docs compatibility symlink is missing: .agents/docs -> ../.claude/docs"
|
|
fail=1
|
|
elif [[ "$(readlink ".agents/docs")" != "../.claude/docs" ]]; then
|
|
echo "error: agent docs compatibility symlink points to $(readlink ".agents/docs"), expected ../.claude/docs"
|
|
fail=1
|
|
fi
|
|
|
|
is_reference_path() {
|
|
local ref="$1"
|
|
case "$ref" in
|
|
*/* | package.json | AGENTS.local.md)
|
|
return 0
|
|
;;
|
|
*)
|
|
return 1
|
|
;;
|
|
esac
|
|
}
|
|
|
|
# TODO: Add circular AGENTS.md include detection if nested agent docs begin
|
|
# referencing each other. Current checks validate file existence only.
|
|
mapfile -t agent_files < <(git ls-files '*AGENTS.md' | sort)
|
|
|
|
for agent_file in "${agent_files[@]}"; do
|
|
agent_dir="$(dirname "$agent_file")"
|
|
while IFS=$'\t' read -r line_number ref; do
|
|
if [[ -z "${line_number:-}" || -z "${ref:-}" ]]; then
|
|
continue
|
|
fi
|
|
if ! is_reference_path "$ref"; then
|
|
continue
|
|
fi
|
|
|
|
candidate="$agent_dir/$ref"
|
|
candidate="${candidate#./}"
|
|
if [[ -e "$candidate" ]]; then
|
|
continue
|
|
fi
|
|
|
|
if [[ "$(basename "$ref")" == "AGENTS.local.md" ]]; then
|
|
echo "warning: $agent_file:$line_number: optional local agent file is not present: $ref"
|
|
continue
|
|
fi
|
|
|
|
echo "error: $agent_file:$line_number: referenced file does not exist: $ref"
|
|
fail=1
|
|
done < <(
|
|
awk '
|
|
/^[[:space:]]*(-[[:space:]]+)?@/ {
|
|
ref = $0
|
|
sub(/^[[:space:]]*(-[[:space:]]+)?@/, "", ref)
|
|
sub(/[[:space:]`)>].*$/, "", ref)
|
|
sub(/[,:;)]+$/, "", ref)
|
|
print FNR "\t" ref
|
|
}
|
|
' "$agent_file"
|
|
)
|
|
done
|
|
|
|
if [[ -f AGENTS.md ]]; then
|
|
root_agent_lines=$(wc -l <AGENTS.md)
|
|
if ((root_agent_lines > 600)); then
|
|
echo "warning: AGENTS.md is $root_agent_lines lines, consider keeping the root guide concise."
|
|
fi
|
|
fi
|
|
|
|
if [[ "$fail" -ne 0 ]]; then
|
|
exit 1
|
|
fi
|
|
|
|
echo "OK: agent docs structure looks valid."
|