mirror of
https://github.com/coder/coder.git
synced 2026-06-02 20:48:20 +00:00
Michael Suchacz
85792d08bc
This PR adds an opinionated harness-engineering layer for agent-driven workflows: a small set of agent-readable docs, mechanical structure checks, structured CI failure summaries, an architecture-lint umbrella, and per-worktree dev-server isolation. The goal is to make local dev, tests, and CI mechanically inspectable by agents without changing app runtime behavior. ## What landed **Agent docs and navigation** - `.claude/docs/OBSERVABILITY.md`, `.claude/docs/DEV_ISOLATION.md`, `.claude/docs/AGENT_FAILURES.md`: task-oriented guides for logs, tracing, Prometheus, dev-server isolation, and a seeded failure catalog. - `AGENTS.md`: added an `Agent navigation` block, then trimmed the file from 375 to 229 lines by migrating duplicated detail into `WORKFLOWS.md`, `GO.md`, `TESTING.md`, and `DATABASE.md`. The user-managed custom-instructions block is preserved. - `.agents/docs`: symlink mirror of `.claude/docs` for agent runtimes that look under `.agents`. **Mechanical checks** - `scripts/check_agents_structure.sh`: validates `@...` references in tracked `AGENTS.md` files and warns when root grows past 600 lines. Wired as `make lint/agents` and into `make lint`. - `scripts/audit-agent-readiness.sh`: report-first audit of harness readiness. Currently `10 ok, 0 warn, 0 fail`. - `scripts/check_architecture.sh` / `make lint/architecture`: umbrella architecture-lint target. Consolidates the existing `check_enterprise_imports.sh` and `check_codersdk_imports.sh` so they run exactly once via the umbrella. Slot is open for new high-confidence rules. **Structured CI failure summaries** - `scripts/playwright-failure-summary.sh`: parses `site/test-results/results.json` and writes Markdown to `$GITHUB_STEP_SUMMARY` on failure. Wired into the `test-e2e` matrix job. - `scripts/go-test-failure-summary.sh`: parses `go test -json` line-delimited output the same way. Wired into `test-go-pg`, `test-go-pg-17`, and `test-go-race-pg` by injecting `gotestsum --jsonfile` in the workflow without touching `Makefile`. JSON also uploaded as a CI artifact on failure. - `site/e2e/playwright.config.ts`: enables `screenshot: only-on-failure`, `trace: retain-on-failure`, JSON reporter, and HTML reporter alongside existing reporters. - `.github/workflows/ci.yaml`: failure artifact uploads for Playwright now use `if: failure()` and predictable names (`playwright-artifacts-<variant>-<sha>`). **Per-worktree dev-server isolation** (`scripts/develop/main.go`) - Deterministic FNV-64a hash of the worktree path produces a port offset in `[0, 1000)` (50 buckets, step 20 to avoid API/proxy overlap across adjacent buckets). - Offset is applied only to defaults; both env vars (`CODER_DEV_PORT`, `CODER_DEV_WEB_PORT`, `CODER_DEV_PROXY_PORT`, `CODER_DEV_PROMETHEUS_PORT`) and CLI flags retain priority. - Hardcoded ports `9090` (embedded Prometheus UI) and `12345` (Delve) are unchanged by design. - Startup banner shows each port's source: `default`, `offset`, or `explicit`. - Unit tests in `scripts/develop/main_test.go` cover determinism, bounds, no-overlap across the four ports, and explicit-skip behavior. - State (`.coderv2/`) was already worktree-isolated via `os.Getwd()`, so no state-dir changes were needed. ## Validation `make lint/agents`, `make lint/architecture`, `make lint/emdash`, `bash scripts/audit-agent-readiness.sh` (10 ok, 0 warn, 0 fail), `shellcheck` on all 5 new scripts, `go test ./scripts/develop/...`, and `js-yaml` parse of `ci.yaml` all pass. Synthetic fixtures verify both failure-summary scripts handle empty/missing input (silent exit 0), ANSI-stripped output, and parent/subtest formatting. ## Known follow-ups (deferred) - Frontend Storybook/Vitest failure summary: lowest-leverage slice of the failure-summary work. Skipping until observed pain. - Architecture lint currently only delegates to existing import checks; new rules (`InTx` outer-store detection, swagger-annotation lint) plug in as needed. - 50 port-offset buckets means two worktree paths can occasionally collide. The DEV_ISOLATION doc tells users to set the relevant env var when this happens. > Mux opened this PR on Mike's behalf.
105 lines
3.3 KiB
Bash
Executable File
105 lines
3.3 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
|
|
# Summarize failed Playwright tests from the JSON reporter output.
|
|
|
|
set -euo pipefail
|
|
# shellcheck source=scripts/lib.sh
|
|
# shellcheck disable=SC1091
|
|
source "$(dirname "${BASH_SOURCE[0]}")/lib.sh"
|
|
cdroot
|
|
|
|
if [[ $# -ne 1 ]]; then
|
|
error "Usage: playwright-failure-summary.sh <results.json>"
|
|
fi
|
|
|
|
results_file=$1
|
|
if [[ ! -f "$results_file" ]]; then
|
|
exit 0
|
|
fi
|
|
|
|
if ! command -v jq >/dev/null; then
|
|
error "jq is required to summarize Playwright failures."
|
|
fi
|
|
|
|
artifact="playwright-artifacts-${MATRIX_VARIANT:-unknown}-${GITHUB_SHA_SHORT:-unknown}"
|
|
|
|
jq -r --arg artifact "$artifact" --arg root "$PROJECT_ROOT" '
|
|
def clean_block:
|
|
tostring
|
|
| gsub("\u001b\\[[0-9;]*[A-Za-z]"; "")
|
|
| gsub("```"; "``");
|
|
def clean_inline:
|
|
tostring | gsub("`"; "");
|
|
def truncate($max):
|
|
if length > $max then .[0:$max] + "..." else . end;
|
|
def failure_status:
|
|
. == "failed" or . == "timedOut" or . == "interrupted";
|
|
def relpath($root):
|
|
if startswith($root + "/") then .[($root | length) + 1:]
|
|
elif startswith("site/") then .
|
|
elif startswith("e2e/") then "site/" + .
|
|
else "site/e2e/" + .
|
|
end;
|
|
def all_specs($titles):
|
|
([$titles[], (.title // empty)] | map(select(. != ""))) as $next_titles
|
|
| (
|
|
.specs[]?
|
|
| . + {
|
|
titlePath: ($next_titles + ([.title // ""] | map(select(. != ""))))
|
|
}
|
|
),
|
|
(.suites[]? | all_specs($next_titles));
|
|
def failure_entries:
|
|
[
|
|
.suites[]?
|
|
| all_specs([]) as $spec
|
|
| $spec.tests[]? as $test
|
|
| select(($test.status // "") != "flaky")
|
|
| select(
|
|
(($test.status // "") == "unexpected")
|
|
or any($test.results[]?; .status | failure_status)
|
|
)
|
|
| ([ $test.results[]? | select(.status | failure_status) ][0]
|
|
// ($test.results[0] // {})) as $result
|
|
| ((($result.error.message // "") | clean_block) as $message
|
|
| (($result.error.stack // "") | clean_block) as $stack
|
|
| {
|
|
file: (($spec.file // "") | relpath($root)),
|
|
line: ($spec.line // 0),
|
|
title: (($spec.titlePath // [$spec.title // ""]) | join(" > ") | clean_inline),
|
|
project: (($test.projectName // "unknown") | clean_inline),
|
|
message: (if $message != "" then $message else $stack end | if . != "" then . else "No error message recorded." end | truncate(600)),
|
|
attachments: ([ $result.attachments[]? | .name // empty | clean_inline ] | unique)
|
|
})
|
|
];
|
|
failure_entries as $entries
|
|
| if ($entries | length) == 0 then
|
|
empty
|
|
else
|
|
(.stats // {}) as $stats
|
|
| ($stats.unexpected // 0) as $stats_failed
|
|
| ([($stats_failed | tonumber), ($entries | length)] | max) as $failed
|
|
| (($stats.expected // 0) + ($stats.unexpected // 0) + ($stats.flaky // 0) + ($stats.skipped // 0)) as $computed_total
|
|
| ($stats.total // $computed_total) as $total
|
|
| [
|
|
"## Playwright failures (\($failed) of \($total))",
|
|
"- Duration: \($stats.duration // 0)ms",
|
|
"- Skipped: \($stats.skipped // 0), Flaky: \($stats.flaky // 0)",
|
|
"- Artifact: `\($artifact)` (download from the run summary)",
|
|
"",
|
|
($entries[]
|
|
| "### \(.file):\(.line)\n"
|
|
+ "- Test: `\(.title)`\n"
|
|
+ "- Project: `\(.project)`\n"
|
|
+ "- Attachments:\n"
|
|
+ (if (.attachments | length) == 0 then
|
|
" - None recorded in artifact `\($artifact)`"
|
|
else
|
|
(.attachments | map(" - `\(.)` in artifact `\($artifact)`") | join("\n"))
|
|
end)
|
|
+ "\n\n```\n\(.message)\n```\n")
|
|
]
|
|
| join("\n")
|
|
end
|
|
' "$results_file" | sed -E $'s/\x1b\[[0-9;]*m//g'
|