# What this does
Dynamic parameters caches the `./terraform/modules` directory for parameter usage. What this PR does is send over this archive to the provisioner when building workspaces.
This allow terraform to skip downloading modules from their registries, a step that takes seconds.
<img width="1223" height="429" alt="Screenshot From 2025-12-29 12-57-52" src="https://github.com/user-attachments/assets/16066e0a-ac79-4296-819d-924f4b0418dc" />
# Wire protocol
The wire protocol reuses the same mechanism used to download the modules `provisoner -> coder`. It splits up large archives into multiple protobuf messages so larger archives can be sent under the message size limit.
# 🚨 Behavior Change (Breaking Change) 🚨
**Before this PR** modules were downloaded on every workspace build. This means unpinned modules always fetched the latest version
**After this PR** modules are cached at template import time, and their versions are effectively pinned for all subsequent workspace builds.
**This is just the protobuf changes for the PR https://github.com/coder/coder/pull/21398**
Moved `UploadFileRequest` from `provisionerd.proto` -> `provisioner.proto`.
Renamed to `FileUpload` because it is now bi-directional.
This **is backwards compatible**. I tested it to confirm the payloads are identical. Types were just renamed and moved around.
```golang
func TestTypeUpgrade(t *testing.T) {
t.Parallel()
x := &proto2.UploadFileRequest{
Type: &proto2.UploadFileRequest_ChunkPiece{
ChunkPiece: &proto.ChunkPiece{
Data: []byte("Hello World!"),
FullDataHash: []byte("Foobar"),
PieceIndex: 42,
},
},
}
data, err := protobuf.Marshal(x)
require.NoError(t, err)
// Exactly the same output
// EhgKDEhlbGxvIFdvcmxkIRIGRm9vYmFyGCo= on `main`
// EhgKDEhlbGxvIFdvcmxkIRIGRm9vYmFyGCo= on this branch
fmt.Println(base64.StdEncoding.EncodeToString(data))
}
```
# What this does
This allows provisioner daemons to download files from `coderd`'s `files` table. This is used to send over cached module files and prevent the need of downloading these modules on each workspace build.
Fixes all our Go file imports to match the preferred spec that we've _mostly_ been using. For example:
```
import (
"context"
"time"
"github.com/prometheus/client_golang/prometheus"
"golang.org/x/xerrors"
"gopkg.in/natefinch/lumberjack.v2"
"cdr.dev/slog/v3"
"github.com/coder/coder/v2/codersdk/agentsdk"
"github.com/coder/serpent"
)
```
3 groups: standard library, 3rd partly libs, Coder libs.
This PR makes the change across the codebase. The PR in the stack above modifies our formatting to maintain this state of affairs, and is a separate PR so it's possible to review that one in detail.
Upgrades to slog v3 which includes a small, but backward incompatible API change to the acceptible call arguments when logging. This change allows us to verify via compile time type checking that arguments are correct and won't cause a panic, as was possible in slog v1, which this replaces (v2 was tagged but never used in coder/coder).
It also updates dependencies that also use slog and were updated.
I've left the `aibridge` dependency as a commit SHA, under the assumption that the team there (cc @pawbana @dannykopping ) will tag and update the dependency soon and on their own schedule.
Other dependencies, I pushed new tags.
Provisioner steps broken into smaller granular actions.
Changes:
- `ExtractArchive` moved to `init` request (was in `configure`)
- Writing `tfstate` moved to `plan` (was in `configure`)
- Moved most plan/apply outputs to `GraphComplete`
Prior to this, every workspace build ran `terraform init` in a fresh
directory. This would mean the `modules` are downloaded fresh. If the
module is not pinned, subsequent workspace builds would have different
modules.
Refactors all Terraform file path logic into a centralized tfpath package. This consolidates all path construction into a single, testable Layout type.
Instead of passing around `string` for directories, pass around the `Layout` which has the file location methods on it.
This PR implements protobuf streaming to handle large module files by:
1. **Streaming large payloads**: When module files exceed the 4MB limit,
they're streamed in chunks using a new UploadFile RPC method
2. **Database storage**: Streamed files are stored in the database and
referenced by hash for deduplication
3. **Backward compatibility**: Small module files continue using the
existing direct payload method
fixes https://github.com/coder/internal/issues/584
Ignore canceled error when sending an acquired job, since dRPC is racy and will sometimes return this error even after successfully sending the job, if the test is quickly finished.
- Adds `testutil.GoleakOptions` and consolidates existing options to
this location
- Pre-emptively adds required ignore for this Dependabot PR to pass CI
https://github.com/coder/coder/pull/16066
Refactors our use of `slogtest` to instantiate a "standard logger" across most of our tests. This standard logger incorporates https://github.com/coder/slog/pull/217 to also ignore database query canceled errors by default, which are a source of low-severity flakes.
Any test that has set non-default `slogtest.Options` is left alone. In particular, `coderdtest` defaults to ignoring all errors. We might consider revisiting that decision now that we have better tools to target the really common flaky Error logs on shutdown.
* fix: separate signals for passive, active, and forced shutdown
`SIGTERM`: Passive shutdown stopping provisioner daemons from accepting new
jobs but waiting for existing jobs to successfully complete.
`SIGINT` (old existing behavior): Notify provisioner daemons to cancel in-flight jobs, wait 5s for jobs to be exited, then force quit.
`SIGKILL`: Untouched from before, will force-quit.
* Revert dramatic signal changes
* Rename
* Fix shutdown behavior for provisioner daemons
* Add test for graceful shutdown
Part of #10532
DRPC transport over yamux and in-mem pipes was previously only used on the provisioner APIs, but now will also be used in tailnet. Moved to subpackage of codersdk to avoid import loops.
fixes#9895
Problem was that provisionerd tries to acquire the next job, and races with shutdown, triggering the assert in the handler. Switches this test case to use the more robust handler.
* chore: add /v2 to import module path
go mod requires semantic versioning with versions greater than 1.x
This was a mechanical update by running:
```
go install github.com/marwan-at-work/mod/cmd/mod@latest
mod upgrade
```
Migrate generated files to import /v2
* Fix gen
* feat: Add bufferring to provisioner job logs
This should improve overall build performance, and especially under load.
It removes the old `id` column on the `provisioner_job_logs` table
and replaces it with an auto-incrementing big integer to preserve order.
Funny enough, we never had to care about order before because inserts
would at minimum be 1ms different. Now they aren't, so the order needs
to be preserved.
* Fix log bufferring
* Fix frontend log streaming
* Fix JS test
* fix: Change uses of t.Cleanup -> defer in test bodies
Mixing t.Cleanup and defer can lead to unexpected order of execution.
* fix: Ensure t.Cleanup is not aborted by require
* chore: Add helper annotations
Steven Masley