shishantbiswas/coder
1
0
Fork 0
mirror of https://github.com/coder/coder.git synced 2026-06-02 20:48:20 +00:00
Code Issues Packages Projects Releases Wiki Activity

chore(Makefile): use go build -o for helper binaries to reduce GOCACHE growth (#24197)

Browse Source 
## Problem

`go run` caches the final linked executable in `~/.cache/go-build`.
Every
helper invocation via `go run ./scripts/<tool>` stores a copy, and
because
the cache key includes build metadata, the same tool accumulates
multiple
cached executables over time. With 12+ helper binaries invoked during
`make gen` and `make pre-commit`, this is a meaningful contributor to
GOCACHE growth.

## Fix

Replace `go run` with `go build -o _gen/bin/<tool>` for 12 repo-local
helper packages (16 Makefile callsites). Each helper is an explicit Make
file target with `$(wildcard *.go)` prerequisites, so `make -j`
serializes
builds correctly instead of racing on shared output paths.

Helpers converted: `apitypings`, `auditdocgen`, `check-scopes`,
`clidocgen`, `dbdump`, `examplegen`, `gensite`, `apikeyscopesgen`,
`metricsdocgen`, `metricsdocgen-scanner`, `modeloptionsgen`, `typegen`.

Left on `go run` (intentionally): `migrate-ci` and `migrate-test`
(CI/test-only, not on common developer paths).

`_gen/` is already in `.gitignore`. The `clean` target removes
`_gen/bin`.

## GOCACHE growth (isolated cache, single `make gen`)

|  | Old (`go run`) | New (`go build -o`) |
|--|----------------|---------------------|
| Total cache size | 2.9 GB | 2.6 GB |
| Cached executables | 11 | 4 |
| Executable bytes | 401 MB | 25 MB |

The 4 remaining executables come from tools outside this change
(`dbgen` and `goimports` from `generate.sh`, plus two `main` binaries
from deferred helpers). Helper binaries now live in `_gen/bin/`
(581 MB, gitignored, cleaned by `make clean`).

## Build time benchmarks

**Source changed** (content hash invalidated, forces recompile):

| Helper | `go run` | `go build -o` + run | Overhead |
|--------|---------|---------------------|----------|
| typegen | 1.50s | 2.03s | +0.52s |
| examplegen | 1.37s | 1.67s | +0.30s |
| apikeyscopesgen | 1.21s | 1.71s | +0.50s |
| modeloptionsgen | 1.23s | 1.64s | +0.41s |

**Repeat invocation** (no source change, the common `make gen` / `make
pre-commit` path):

| Helper | `go run` (cache lookup) | Cached binary | Speedup |
|--------|------------------------|---------------|---------|
| typegen | 0.346s | 0.037s | 9.4x |
| examplegen | 0.368s | 0.037s | 9.9x |
| modeloptionsgen | 0.342s | 0.021s | 16.3x |
| apikeyscopesgen | 0.298s | 0.030s | 9.9x |

When source changes, `go build -o` is 0.3-0.5s slower per helper (it
writes a local binary instead of caching in GOCACHE). On repeat runs
(the common path), the pre-built binary is 10-16x faster because
`go run` still does a staleness check while the binary just executes.

> This PR was authored by Mux on behalf of Mike.
This commit is contained in:
Michael Suchacz
2026-04-09 16:04:06 +02:00
committed by GitHub
parent fb0ed1162b
commit f8e8f979a2
1 changed files with 95 additions and 41 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Makefile
+95 -41
View File
@@ -91,6 +91,59 @@ define atomic_write
mv "$$tmpfile" "$@" && rm -rf "$$tmpdir"
endef
# Helper binary targets. Built with go build -o to avoid caching
# link-stage executables in GOCACHE. Each binary is a real Make
# target so parallel -j builds serialize correctly instead of
# racing on the same output path.
_gen/bin/apitypings: $(wildcard scripts/apitypings/*.go) | _gen
@mkdir -p _gen/bin
go build -o $@ ./scripts/apitypings
_gen/bin/auditdocgen: $(wildcard scripts/auditdocgen/*.go) | _gen
@mkdir -p _gen/bin
go build -o $@ ./scripts/auditdocgen
_gen/bin/check-scopes: $(wildcard scripts/check-scopes/*.go) | _gen
@mkdir -p _gen/bin
go build -o $@ ./scripts/check-scopes
_gen/bin/clidocgen: $(wildcard scripts/clidocgen/*.go) | _gen
@mkdir -p _gen/bin
go build -o $@ ./scripts/clidocgen
_gen/bin/dbdump: $(wildcard coderd/database/gen/dump/*.go) | _gen
@mkdir -p _gen/bin
go build -o $@ ./coderd/database/gen/dump
_gen/bin/examplegen: $(wildcard scripts/examplegen/*.go) | _gen
@mkdir -p _gen/bin
go build -o $@ ./scripts/examplegen
_gen/bin/gensite: $(wildcard scripts/gensite/*.go) | _gen
@mkdir -p _gen/bin
go build -o $@ ./scripts/gensite
_gen/bin/apikeyscopesgen: $(wildcard scripts/apikeyscopesgen/*.go) | _gen
@mkdir -p _gen/bin
go build -o $@ ./scripts/apikeyscopesgen
_gen/bin/metricsdocgen: $(wildcard scripts/metricsdocgen/*.go) | _gen
@mkdir -p _gen/bin
go build -o $@ ./scripts/metricsdocgen
_gen/bin/metricsdocgen-scanner: $(wildcard scripts/metricsdocgen/scanner/*.go) | _gen
@mkdir -p _gen/bin
go build -o $@ ./scripts/metricsdocgen/scanner
_gen/bin/modeloptionsgen: $(wildcard scripts/modeloptionsgen/*.go) | _gen
@mkdir -p _gen/bin
go build -o $@ ./scripts/modeloptionsgen
_gen/bin/typegen: $(wildcard scripts/typegen/*.go) | _gen
@mkdir -p _gen/bin
go build -o $@ ./scripts/typegen
# Shared temp directory for atomic writes. Lives at the project root
# so all targets share the same filesystem, and is gitignored.
# Order-only prerequisite: recipes that need it depend on | _gen
@@ -201,6 +254,7 @@ endif
clean:
rm -rf build/ site/build/ site/out/
rm -rf _gen/bin
mkdir -p build/
git restore site/out/
.PHONY: clean
@@ -654,8 +708,8 @@ lint/go:
go tool github.com/coder/paralleltestctx/cmd/paralleltestctx -custom-funcs="testutil.Context" ./...
.PHONY: lint/go
lint/examples:
go run ./scripts/examplegen/main.go -lint
lint/examples: | _gen/bin/examplegen
_gen/bin/examplegen -lint
.PHONY: lint/examples
# Use shfmt to determine the shell files, takes editorconfig into consideration.
@@ -693,8 +747,8 @@ lint/actions/zizmor:
.PHONY: lint/actions/zizmor
# Verify api_key_scope enum contains all RBAC <resource>:<action> values.
lint/check-scopes: coderd/database/dump.sql
go run ./scripts/check-scopes
lint/check-scopes: coderd/database/dump.sql | _gen/bin/check-scopes
_gen/bin/check-scopes
.PHONY: lint/check-scopes
# Verify migrations do not hardcode the public schema.
@@ -734,8 +788,8 @@ lint/typos: build/typos-$(TYPOS_VERSION)
# The pre-push hook is allowlisted, see scripts/githooks/pre-push.
#
# pre-commit uses two phases: gen+fmt first, then lint+build. This
# avoids races where gen's `go run` creates temporary .go files that
# lint's find-based checks pick up. Within each phase, targets run in
# avoids races where gen creates temporary .go files that lint's
# find-based checks pick up. Within each phase, targets run in
# parallel via -j. It fails if any tracked files have unstaged
# changes afterward.
@@ -949,8 +1003,8 @@ gen/mark-fresh:
# Runs migrations to output a dump of the database schema after migrations are
# applied.
coderd/database/dump.sql: coderd/database/gen/dump/main.go $(wildcard coderd/database/migrations/*.sql)
go run ./coderd/database/gen/dump/main.go
coderd/database/dump.sql: coderd/database/gen/dump/main.go $(wildcard coderd/database/migrations/*.sql) | _gen/bin/dbdump
_gen/bin/dbdump
touch "$@"
# Generates Go code for querying the database.
@@ -1067,88 +1121,88 @@ enterprise/aibridged/proto/aibridged.pb.go: enterprise/aibridged/proto/aibridged
--go-drpc_opt=paths=source_relative \
./enterprise/aibridged/proto/aibridged.proto
site/src/api/typesGenerated.ts: site/node_modules/.installed $(wildcard scripts/apitypings/*) $(shell find ./codersdk $(FIND_EXCLUSIONS) -type f -name '*.go') | _gen
$(call atomic_write,go run -C ./scripts/apitypings main.go,./scripts/biome_format.sh)
site/src/api/typesGenerated.ts: site/node_modules/.installed $(wildcard scripts/apitypings/*) $(shell find ./codersdk $(FIND_EXCLUSIONS) -type f -name '*.go') | _gen _gen/bin/apitypings
$(call atomic_write,_gen/bin/apitypings,./scripts/biome_format.sh)
site/e2e/provisionerGenerated.ts: site/node_modules/.installed provisionerd/proto/provisionerd.pb.go provisionersdk/proto/provisioner.pb.go
(cd site/ && pnpm run gen:provisioner)
touch "$@"
site/src/theme/icons.json: site/node_modules/.installed $(wildcard scripts/gensite/*) $(wildcard site/static/icon/*) | _gen
site/src/theme/icons.json: site/node_modules/.installed $(wildcard scripts/gensite/*) $(wildcard site/static/icon/*) | _gen _gen/bin/gensite
tmpdir=$$(mktemp -d -p _gen) && tmpfile=$$(realpath "$$tmpdir")/$(notdir $@) && \
go run ./scripts/gensite/ -icons "$$tmpfile" && \
_gen/bin/gensite -icons "$$tmpfile" && \
./scripts/biome_format.sh "$$tmpfile" && \
mv "$$tmpfile" "$@" && rm -rf "$$tmpdir"
examples/examples.gen.json: scripts/examplegen/main.go examples/examples.go $(shell find ./examples/templates) | _gen
$(call atomic_write,go run ./scripts/examplegen/main.go)
examples/examples.gen.json: scripts/examplegen/main.go examples/examples.go $(shell find ./examples/templates) | _gen _gen/bin/examplegen
$(call atomic_write,_gen/bin/examplegen)
coderd/rbac/object_gen.go: scripts/typegen/rbacobject.gotmpl scripts/typegen/main.go coderd/rbac/object.go coderd/rbac/policy/policy.go | _gen
$(call atomic_write,go run ./scripts/typegen/main.go rbac object)
coderd/rbac/object_gen.go: scripts/typegen/rbacobject.gotmpl scripts/typegen/main.go coderd/rbac/object.go coderd/rbac/policy/policy.go | _gen _gen/bin/typegen
$(call atomic_write,_gen/bin/typegen rbac object)
touch "$@"
# NOTE: depends on object_gen.go because `go run` compiles
# coderd/rbac which includes it.
# NOTE: depends on object_gen.go because the generator build
# compiles coderd/rbac which includes it.
coderd/rbac/scopes_constants_gen.go: scripts/typegen/scopenames.gotmpl scripts/typegen/main.go coderd/rbac/policy/policy.go \
coderd/rbac/object_gen.go | _gen
coderd/rbac/object_gen.go | _gen _gen/bin/typegen
# Write to a temp file first to avoid truncating the package
# during build since the generator imports the rbac package.
$(call atomic_write,go run ./scripts/typegen/main.go rbac scopenames)
$(call atomic_write,_gen/bin/typegen rbac scopenames)
touch "$@"
# NOTE: depends on object_gen.go and scopes_constants_gen.go because
# `go run` compiles coderd/rbac which includes both.
# the generator build compiles coderd/rbac which includes both.
codersdk/rbacresources_gen.go: scripts/typegen/codersdk.gotmpl scripts/typegen/main.go coderd/rbac/object.go coderd/rbac/policy/policy.go \
coderd/rbac/object_gen.go coderd/rbac/scopes_constants_gen.go | _gen
coderd/rbac/object_gen.go coderd/rbac/scopes_constants_gen.go | _gen _gen/bin/typegen
# Write to a temp file to avoid truncating the target, which
# would break the codersdk package and any parallel build targets.
$(call atomic_write,go run scripts/typegen/main.go rbac codersdk)
$(call atomic_write,_gen/bin/typegen rbac codersdk)
touch "$@"
# NOTE: depends on object_gen.go and scopes_constants_gen.go because
# `go run` compiles coderd/rbac which includes both.
# the generator build compiles coderd/rbac which includes both.
codersdk/apikey_scopes_gen.go: scripts/apikeyscopesgen/main.go coderd/rbac/scopes_catalog.go coderd/rbac/scopes.go \
coderd/rbac/object_gen.go coderd/rbac/scopes_constants_gen.go | _gen
coderd/rbac/object_gen.go coderd/rbac/scopes_constants_gen.go | _gen _gen/bin/apikeyscopesgen
# Generate SDK constants for external API key scopes.
$(call atomic_write,go run ./scripts/apikeyscopesgen)
$(call atomic_write,_gen/bin/apikeyscopesgen)
touch "$@"
# NOTE: depends on object_gen.go and scopes_constants_gen.go because
# `go run` compiles coderd/rbac which includes both.
# the generator build compiles coderd/rbac which includes both.
site/src/api/rbacresourcesGenerated.ts: site/node_modules/.installed scripts/typegen/codersdk.gotmpl scripts/typegen/main.go coderd/rbac/object.go coderd/rbac/policy/policy.go \
coderd/rbac/object_gen.go coderd/rbac/scopes_constants_gen.go | _gen
$(call atomic_write,go run scripts/typegen/main.go rbac typescript,./scripts/biome_format.sh)
coderd/rbac/object_gen.go coderd/rbac/scopes_constants_gen.go | _gen _gen/bin/typegen
$(call atomic_write,_gen/bin/typegen rbac typescript,./scripts/biome_format.sh)
site/src/api/countriesGenerated.ts: site/node_modules/.installed scripts/typegen/countries.tstmpl scripts/typegen/main.go codersdk/countries.go | _gen
$(call atomic_write,go run scripts/typegen/main.go countries,./scripts/biome_format.sh)
site/src/api/countriesGenerated.ts: site/node_modules/.installed scripts/typegen/countries.tstmpl scripts/typegen/main.go codersdk/countries.go | _gen _gen/bin/typegen
$(call atomic_write,_gen/bin/typegen countries,./scripts/biome_format.sh)
site/src/api/chatModelOptionsGenerated.json: scripts/modeloptionsgen/main.go codersdk/chats.go | _gen
$(call atomic_write,go run ./scripts/modeloptionsgen/main.go | tail -n +2,./scripts/biome_format.sh)
site/src/api/chatModelOptionsGenerated.json: scripts/modeloptionsgen/main.go codersdk/chats.go | _gen _gen/bin/modeloptionsgen
$(call atomic_write,_gen/bin/modeloptionsgen | tail -n +2,./scripts/biome_format.sh)
scripts/metricsdocgen/generated_metrics: $(GO_SRC_FILES) | _gen
$(call atomic_write,go run ./scripts/metricsdocgen/scanner)
scripts/metricsdocgen/generated_metrics: $(GO_SRC_FILES) | _gen _gen/bin/metricsdocgen-scanner
$(call atomic_write,_gen/bin/metricsdocgen-scanner)
docs/admin/integrations/prometheus.md: node_modules/.installed scripts/metricsdocgen/main.go scripts/metricsdocgen/metrics scripts/metricsdocgen/generated_metrics | _gen
docs/admin/integrations/prometheus.md: node_modules/.installed scripts/metricsdocgen/main.go scripts/metricsdocgen/metrics scripts/metricsdocgen/generated_metrics | _gen _gen/bin/metricsdocgen
tmpdir=$$(mktemp -d -p _gen) && tmpfile=$$(realpath "$$tmpdir")/$(notdir $@) && cp "$@" "$$tmpfile" && \
go run scripts/metricsdocgen/main.go --prometheus-doc-file="$$tmpfile" && \
_gen/bin/metricsdocgen --prometheus-doc-file="$$tmpfile" && \
pnpm exec markdownlint-cli2 --fix "$$tmpfile" && \
pnpm exec markdown-table-formatter "$$tmpfile" && \
mv "$$tmpfile" "$@" && rm -rf "$$tmpdir"
docs/reference/cli/index.md: node_modules/.installed scripts/clidocgen/main.go examples/examples.gen.json $(GO_SRC_FILES) | _gen
docs/reference/cli/index.md: node_modules/.installed scripts/clidocgen/main.go examples/examples.gen.json $(GO_SRC_FILES) | _gen _gen/bin/clidocgen
tmpdir=$$(mktemp -d -p _gen) && \
tmpdir=$$(realpath "$$tmpdir") && \
mkdir -p "$$tmpdir/docs/reference/cli" && \
cp docs/manifest.json "$$tmpdir/docs/manifest.json" && \
CI=true DOCS_DIR="$$tmpdir/docs" go run ./scripts/clidocgen && \
CI=true DOCS_DIR="$$tmpdir/docs" _gen/bin/clidocgen && \
pnpm exec markdownlint-cli2 --fix "$$tmpdir/docs/reference/cli/*.md" && \
pnpm exec markdown-table-formatter "$$tmpdir/docs/reference/cli/*.md" && \
for f in "$$tmpdir/docs/reference/cli/"*.md; do mv "$$f" "docs/reference/cli/$$(basename "$$f")"; done && \
rm -rf "$$tmpdir"
docs/admin/security/audit-logs.md: node_modules/.installed coderd/database/querier.go scripts/auditdocgen/main.go enterprise/audit/table.go coderd/rbac/object_gen.go | _gen
docs/admin/security/audit-logs.md: node_modules/.installed coderd/database/querier.go scripts/auditdocgen/main.go enterprise/audit/table.go coderd/rbac/object_gen.go | _gen _gen/bin/auditdocgen
tmpdir=$$(mktemp -d -p _gen) && tmpfile=$$(realpath "$$tmpdir")/$(notdir $@) && cp "$@" "$$tmpfile" && \
go run scripts/auditdocgen/main.go --audit-doc-file="$$tmpfile" && \
_gen/bin/auditdocgen --audit-doc-file="$$tmpfile" && \
pnpm exec markdownlint-cli2 --fix "$$tmpfile" && \
pnpm exec markdown-table-formatter "$$tmpfile" && \
mv "$$tmpfile" "$@" && rm -rf "$$tmpdir"