Files
Michael Suchacz 85792d08bc feat: add harness engineering layer for agent workflows (#24791)
This PR adds an opinionated harness-engineering layer for agent-driven
workflows: a small set of agent-readable docs, mechanical structure
checks, structured CI failure summaries, an architecture-lint umbrella,
and per-worktree dev-server isolation. The goal is to make local dev,
tests, and CI mechanically inspectable by agents without changing app
runtime behavior.

## What landed

**Agent docs and navigation**
- `.claude/docs/OBSERVABILITY.md`, `.claude/docs/DEV_ISOLATION.md`,
`.claude/docs/AGENT_FAILURES.md`: task-oriented guides for logs,
tracing, Prometheus, dev-server isolation, and a seeded failure catalog.
- `AGENTS.md`: added an `Agent navigation` block, then trimmed the file
from 375 to 229 lines by migrating duplicated detail into
`WORKFLOWS.md`, `GO.md`, `TESTING.md`, and `DATABASE.md`. The
user-managed custom-instructions block is preserved.
- `.agents/docs`: symlink mirror of `.claude/docs` for agent runtimes
that look under `.agents`.

**Mechanical checks**
- `scripts/check_agents_structure.sh`: validates `@...` references in
tracked `AGENTS.md` files and warns when root grows past 600 lines.
Wired as `make lint/agents` and into `make lint`.
- `scripts/audit-agent-readiness.sh`: report-first audit of harness
readiness. Currently `10 ok, 0 warn, 0 fail`.
- `scripts/check_architecture.sh` / `make lint/architecture`: umbrella
architecture-lint target. Consolidates the existing
`check_enterprise_imports.sh` and `check_codersdk_imports.sh` so they
run exactly once via the umbrella. Slot is open for new high-confidence
rules.

**Structured CI failure summaries**
- `scripts/playwright-failure-summary.sh`: parses
`site/test-results/results.json` and writes Markdown to
`$GITHUB_STEP_SUMMARY` on failure. Wired into the `test-e2e` matrix job.
- `scripts/go-test-failure-summary.sh`: parses `go test -json`
line-delimited output the same way. Wired into `test-go-pg`,
`test-go-pg-17`, and `test-go-race-pg` by injecting `gotestsum
--jsonfile` in the workflow without touching `Makefile`. JSON also
uploaded as a CI artifact on failure.
- `site/e2e/playwright.config.ts`: enables `screenshot:
only-on-failure`, `trace: retain-on-failure`, JSON reporter, and HTML
reporter alongside existing reporters.
- `.github/workflows/ci.yaml`: failure artifact uploads for Playwright
now use `if: failure()` and predictable names
(`playwright-artifacts-<variant>-<sha>`).

**Per-worktree dev-server isolation** (`scripts/develop/main.go`)
- Deterministic FNV-64a hash of the worktree path produces a port offset
in `[0, 1000)` (50 buckets, step 20 to avoid API/proxy overlap across
adjacent buckets).
- Offset is applied only to defaults; both env vars (`CODER_DEV_PORT`,
`CODER_DEV_WEB_PORT`, `CODER_DEV_PROXY_PORT`,
`CODER_DEV_PROMETHEUS_PORT`) and CLI flags retain priority.
- Hardcoded ports `9090` (embedded Prometheus UI) and `12345` (Delve)
are unchanged by design.
- Startup banner shows each port's source: `default`, `offset`, or
`explicit`.
- Unit tests in `scripts/develop/main_test.go` cover determinism,
bounds, no-overlap across the four ports, and explicit-skip behavior.
- State (`.coderv2/`) was already worktree-isolated via `os.Getwd()`, so
no state-dir changes were needed.

## Validation

`make lint/agents`, `make lint/architecture`, `make lint/emdash`, `bash
scripts/audit-agent-readiness.sh` (10 ok, 0 warn, 0 fail), `shellcheck`
on all 5 new scripts, `go test ./scripts/develop/...`, and `js-yaml`
parse of `ci.yaml` all pass. Synthetic fixtures verify both
failure-summary scripts handle empty/missing input (silent exit 0),
ANSI-stripped output, and parent/subtest formatting.

## Known follow-ups (deferred)

- Frontend Storybook/Vitest failure summary: lowest-leverage slice of
the failure-summary work. Skipping until observed pain.
- Architecture lint currently only delegates to existing import checks;
new rules (`InTx` outer-store detection, swagger-annotation lint) plug
in as needed.
- 50 port-offset buckets means two worktree paths can occasionally
collide. The DEV_ISOLATION doc tells users to set the relevant env var
when this happens.

> Mux opened this PR on Mike's behalf.
2026-05-11 17:27:29 +02:00

11 KiB

Development Workflows and Guidelines

Quick Start Checklist for New Features

Before Starting

  • Run git pull to ensure you're on latest code
  • Check if feature touches database - you'll need migrations
  • Check if feature touches audit logs - update enterprise/audit/table.go

Development Server

Starting Development Mode

  • Use ./scripts/develop.sh to start Coder in development mode
  • This automatically builds and runs with --dev flag and proper access URL
  • ⚠️ Do NOT manually run make build && ./coder server --dev - use the script instead

Development Workflow

  1. Always start with the development script: ./scripts/develop.sh
  2. Make changes to your code
  3. The script will automatically rebuild and restart as needed
  4. Access the development server at the URL provided by the script

Code Style Guidelines

Go Style

  • Follow Effective Go and Go's Code Review Comments
  • Create packages when used during implementation
  • Validate abstractions against implementations
  • Test packages: Use package_test naming (e.g., identityprovider_test) for black-box testing

Error Handling

  • Use descriptive error messages
  • Wrap errors with context
  • Propagate errors appropriately
  • Use proper error types
  • Pattern: xerrors.Errorf("failed to X: %w", err)

Naming Conventions

  • Names MUST tell what code does, not how it's implemented or its history
  • Follow Go and TypeScript naming conventions
  • When changing code, never document the old behavior or the behavior change
  • NEVER use implementation details in names (e.g., "ZodValidator", "MCPWrapper", "JSONParser")
  • NEVER use temporal/historical context in names (e.g., "LegacyHandler", "UnifiedTool", "ImprovedInterface", "EnhancedParser")
  • NEVER use pattern names unless they add clarity (e.g., prefer "Tool" over "ToolFactory")
  • Abbreviate only when obvious

Comments

  • Document exported functions, types, and non-obvious logic
  • Follow JSDoc format for TypeScript
  • Use godoc format for Go code

Database Migration Workflows

Migration Guidelines

  1. Create migration files:

    • Location: coderd/database/migrations/
    • Format: {number}_{description}.{up|down}.sql
    • Number must be unique and sequential
    • Always include both up and down migrations
  2. Use helper scripts:

    • ./coderd/database/migrations/create_migration.sh "migration name" - Creates new migration files
    • ./coderd/database/migrations/fix_migration_numbers.sh - Renumbers migrations to avoid conflicts
    • ./coderd/database/migrations/create_fixture.sh "fixture name" - Creates test fixtures for migrations
  3. Update database queries:

    • MUST DO: Any changes to database - adding queries, modifying queries should be done in the coderd/database/queries/*.sql files
    • MUST DO: Queries are grouped in files relating to context - e.g. prebuilds.sql, users.sql, oauth2.sql
    • After making changes to any coderd/database/queries/*.sql files you must run make gen to generate respective ORM changes
  4. Handle nullable fields:

    • Use sql.NullString, sql.NullBool, etc. for optional database fields
    • Set .Valid = true when providing values
  5. Audit table updates:

    • If adding fields to auditable types, update enterprise/audit/table.go
    • Add each new field with appropriate action (ActionTrack, ActionIgnore, ActionSecret)
    • Run make gen to verify no audit errors

Database Generation Process

  1. Modify SQL files in coderd/database/queries/
  2. Run make gen
  3. If errors about audit table, update enterprise/audit/table.go
  4. Run make gen again
  5. Run make lint to catch any remaining issues

API Development Workflow

Adding New API Endpoints

  1. Define types in codersdk/ package
  2. Add handler in appropriate coderd/ file
  3. Register route in coderd/coderd.go
  4. Add tests in coderd/*_test.go files
  5. Update OpenAPI by running make gen

API Design Guardrails

  • Add swagger annotations when introducing new HTTP endpoints. Do this in the same change as the handler so the docs do not get missed before release.
  • For user-scoped or resource-scoped routes, prefer path parameters over query parameters when that matches existing route patterns.
  • For experimental or unstable API paths, skip public doc generation with // @x-apidocgen {"skip": true} after the @Router annotation. This keeps them out of the published API reference until they stabilize.

Testing Workflows

Test Execution

  • Run full test suite: make test
  • Run specific test: make test RUN=TestFunctionName
  • Run with race detector: make test-race
  • Run end-to-end tests: make test-e2e

Test Development

  • Use table-driven tests for comprehensive coverage
  • Mock external dependencies
  • Test both positive and negative cases
  • Use testutil.WaitLong for timeouts in tests
  • Always use t.Parallel() in tests

Git Workflow

Git Hooks

You MUST install and use the git hooks. NEVER bypass them with --no-verify. Skipping hooks wastes CI cycles and is unacceptable.

The first run will be slow as caches warm up. Consecutive runs are significantly faster (often 10x) thanks to Go build cache, generated file timestamps, and warm node_modules. This is NOT a reason to skip them. Wait for hooks to complete before proceeding, no matter how long they take.

git config core.hooksPath scripts/githooks

Two hooks run automatically:

  • pre-commit: Classifies staged files by type and runs either the full make pre-commit or the lightweight make pre-commit-light depending on whether Go, TypeScript, SQL, proto, or Makefile changes are present. Falls back to the full target when CODER_HOOK_RUN_ALL=1 is set. A markdown-only commit takes seconds; a Go change takes several minutes.
  • pre-push: Classifies changed files (vs remote branch or merge-base) and runs make pre-push when Go, TypeScript, SQL, proto, or Makefile changes are detected. Skips tests entirely for lightweight changes. Allowlisted in scripts/githooks/pre-push. Runs only for developers who opt in. Falls back to make pre-push when the diff range can't be determined or CODER_HOOK_RUN_ALL=1 is set. Allow at least 15 minutes for a full run.

git commit and git push will appear to hang while hooks run. This is normal. Do not interrupt, retry, or reduce the timeout.

NEVER run git config core.hooksPath to change or disable hooks.

If a hook fails, fix the issue and retry. Do not work around the failure by skipping the hook.

Working on PR branches

When working on an existing PR branch:

git fetch origin
git checkout branch-name
git pull origin branch-name

Then make your changes and push normally. Don't use git push --force unless the user specifically asks for it.

Commit Style

Format: type(scope): message. See CONTRIBUTING.md for full rules. PR titles are linted in CI.

  • Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore, revert
  • Scopes must be a real path (directory or file stem) containing all changed files
  • Omit scope if changes span multiple top-level directories
  • Keep message titles concise (~70 characters)
  • Use imperative, present tense in commit titles

Code Navigation and Investigation

IMPORTANT: Always use LSP tools for code navigation and understanding. These tools provide accurate, real-time analysis of the codebase and should be your first choice for code investigation.

Go LSP Tools (for backend code)

  1. Find function definitions (USE THIS FREQUENTLY):

    • mcp__go-language-server__definition symbolName
    • Example: mcp__go-language-server__definition getOAuth2ProviderAppAuthorize
    • Quickly jump to function implementations across packages
  2. Find symbol references (ESSENTIAL FOR UNDERSTANDING IMPACT):

    • mcp__go-language-server__references symbolName
    • Locate all usages of functions, types, or variables
    • Critical for refactoring and understanding data flow
  3. Get symbol information:

    • mcp__go-language-server__hover filePath line column
    • Get type information and documentation at specific positions

TypeScript LSP Tools (for frontend code in site/)

  1. Find component/function definitions (USE THIS FREQUENTLY):

    • mcp__typescript-language-server__definition symbolName
    • Example: mcp__typescript-language-server__definition LoginPage
    • Quickly navigate to React components, hooks, and utility functions
  2. Find symbol references (ESSENTIAL FOR UNDERSTANDING IMPACT):

    • mcp__typescript-language-server__references symbolName
    • Locate all usages of components, types, or functions
    • Critical for refactoring React components and understanding prop usage
  3. Get type information:

    • mcp__typescript-language-server__hover filePath line column
    • Get TypeScript type information and JSDoc documentation
  4. Rename symbols safely:

    • mcp__typescript-language-server__rename_symbol filePath line column newName
    • Rename components, props, or functions across the entire codebase
  5. Check for TypeScript errors:

    • mcp__typescript-language-server__diagnostics filePath
    • Get compilation errors and warnings for a specific file

Investigation Strategy (LSP-First Approach)

Backend Investigation (Go)

  1. Start with route registration in coderd/coderd.go to understand API endpoints
  2. Use Go LSP definition lookup to trace from route handlers to actual implementations
  3. Use Go LSP references to understand how functions are called throughout the codebase
  4. Follow the middleware chain using LSP tools to understand request processing flow
  5. Check test files for expected behavior and error patterns

Frontend Investigation (TypeScript/React)

  1. Start with route definitions in site/src/App.tsx or router configuration
  2. Use TypeScript LSP definition to navigate to React components and hooks
  3. Use TypeScript LSP references to find all component usages and prop drilling
  4. Follow the component hierarchy using LSP tools to understand data flow
  5. Check for TypeScript errors with diagnostics before making changes
  6. Examine test files (.test.tsx) for component behavior and expected props

Troubleshooting Development Issues

Common Issues

  1. Development server won't start - Use ./scripts/develop.sh instead of manual commands
  2. Database migration errors - Check migration file format and use helper scripts
  3. Audit table errors - Update enterprise/audit/table.go with new fields
  4. OAuth2 compliance issues - Ensure RFC-compliant error responses

Debug Commands

  • Check linting: make lint
  • Generate code: make gen
  • Clean build: make clean

Development Environment Setup

Prerequisites

  • Go (version specified in go.mod)
  • Node.js and pnpm for frontend development
  • PostgreSQL for database testing
  • Docker for containerized testing

First Time Setup

  1. Clone the repository
  2. Run ./scripts/develop.sh to start development server
  3. Access the development URL provided
  4. Create admin user as prompted
  5. Begin development