ci: add Go test flake detector workflow (#25667)

Adds a `flake-go` workflow that hunts for ordering-dependent and racy Go
tests on pull requests. The workflow runs only on PRs (cancelling
earlier runs on new commits) and skips test execution when no Go test
files changed.

A single `flake_go` job uses
[coder/whichtests](https://github.com/coder/whichtests) with
`--coalesce` to compute the directly-modified `Test*` functions from the
PR diff and emit them as one target row. The same job then runs those
selected tests on a deliberately resource-constrained 4-vCPU runner with
4x parallelism oversubscription, `-count=25`, and `-shuffle=on` to
amplify contention and surface flakes.

Pinned at
[coder/whichtests@ec33bab](https://github.com/coder/whichtests/commit/ec33bab1ec04cd86beb7a61a069db4463dba63f5).

Reuses the `test-go-pg` composite (with its new `run-regex`,
`test-shuffle`, and `gotestsum-json-file` inputs) and the
`go-test-failure-report` composite, both introduced on the base branch
(#25670), so this workflow shares one implementation of the gotestsum +
failure-report path with the existing CI jobs.

`Makefile` adds `TEST_SHUFFLE` support and single-quotes `RUN` so
whichtests' regex survives shell parsing.

Stacked on top of #25670.

Demo @
https://github.com/coder/coder/actions/runs/26494322649/job/78018779381?pr=25667

Closes CODAGT-381
This commit is contained in:
Ethan
2026-05-28 12:35:37 +10:00
committed by GitHub
parent 9d37f63fbd
commit ca7f07142e
2 changed files with 91 additions and 1 deletions
+82
View File
@@ -0,0 +1,82 @@
name: flake-go
on:
pull_request:
workflow_dispatch:
inputs:
base_sha:
description: "Base commit to diff against. Defaults to merge-base against origin/main."
required: false
type: string
head_sha:
description: "Head commit to analyze. Defaults to the checked out HEAD."
required: false
type: string
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
flake_go:
name: Flake Check
runs-on: ${{ github.repository_owner == 'coder' && 'depot-ubuntu-22.04-4' || 'ubuntu-latest' }}
timeout-minutes: 20
steps:
- name: Harden Runner
uses: step-security/harden-runner@f808768d1510423e83855289c910610ca9b43176 # v2.17.0
with:
egress-policy: audit
- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name || github.repository }}
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.event.inputs.head_sha || github.sha }}
fetch-depth: 0
persist-credentials: false
- name: Setup Go
uses: ./.github/actions/setup-go
- name: Install whichtests
shell: bash
run: ./.github/scripts/retry.sh -- go install github.com/coder/whichtests@ec33bab1ec04cd86beb7a61a069db4463dba63f5
- name: Select changed tests
id: selector
shell: bash
run: |
set -euo pipefail
whichtests \
--repo-root . \
--github-actions \
--coalesce \
--out-matrix "$RUNNER_TEMP/flake-matrix.json"
- name: Setup Terraform
if: ${{ fromJSON(steps.selector.outputs.matrix).include[0] != null }}
uses: ./.github/actions/setup-tf
- name: Run targeted Go flake checks
id: flake_check
if: ${{ fromJSON(steps.selector.outputs.matrix).include[0] != null }}
uses: ./.github/actions/test-go-pg
with:
postgres-version: "13"
test-parallelism-packages: "4"
test-parallelism-tests: "16"
test-count: "25"
test-packages: ${{ fromJSON(steps.selector.outputs.matrix).include[0].package }}
run-regex: ${{ fromJSON(steps.selector.outputs.matrix).include[0].run_regex }}
test-shuffle: "on"
gotestsum-json-file: default
- name: Publish Go test failure report
if: failure() && steps.flake_check.outcome == 'failure' && github.actor != 'dependabot[bot]' && runner.os == 'Linux' && (github.event_name != 'pull_request' || !github.event.pull_request.head.repo.fork)
uses: ./.github/actions/go-test-failure-report
with:
artifact-name: go-test-failures-${{ github.job }}-${{ github.sha }}
+9 -1
View File
@@ -1446,8 +1446,16 @@ ifdef TEST_SHORT
GOTEST_FLAGS += -short
endif
# RUN is single-quoted for the shell so regex metacharacters survive make.
# Embedded single quotes are not supported; whichtests only emits RUN values
# built from ASCII test names so generated regexes stay within this contract.
ifdef RUN
GOTEST_FLAGS += -run $(RUN)
GOTEST_FLAGS += -run '$(RUN)'
endif
# TEST_SHUFFLE values must be off, on, or an integer seed.
ifdef TEST_SHUFFLE
GOTEST_FLAGS += -shuffle=$(TEST_SHUFFLE)
endif
ifdef TEST_CPUPROFILE