This PR implements protobuf streaming to handle large module files by:
1. **Streaming large payloads**: When module files exceed the 4MB limit,
they're streamed in chunks using a new UploadFile RPC method
2. **Database storage**: Streamed files are stored in the database and
referenced by hash for deduplication
3. **Backward compatibility**: Small module files continue using the
existing direct payload method
Closes https://github.com/coder/internal/issues/369
We can't know whether a replacement (i.e. drift of terraform state
leading to a resource needing to be deleted/recreated) will take place
apriori; we can only detect it at `plan` time, because the provider
decides whether a resource must be replaced and it cannot be inferred
through static analysis of the template.
**This is likely to be the most common gotcha with using prebuilds,
since it requires a slight template modification to use prebuilds
effectively**, so let's head this off before it's an issue for
customers.
Drift details will now be logged in the workspace build logs:
Plus a notification will be sent to template admins when this situation
arises:
A new metric - `coderd_prebuilt_workspaces_resource_replacements_total`
- will also increment each time a workspace encounters replacements.
We only track _that_ a resource replacement occurred, not how many. Just
one is enough to ruin a prebuild, but we can't know apriori which
replacement would cause this.
For example, say we have 2 replacements: a `docker_container` and a
`null_resource`; we don't know which one might
cause an issue (or indeed if either would), so we just track the
replacement.
---------
Signed-off-by: Danny Kopping <dannykopping@gmail.com>
- Update go.mod to use Go 1.24.1
- Update GitHub Actions setup-go action to use Go 1.24.1
- Fix linting issues with golangci-lint by:
- Updating to golangci-lint v1.57.1 (more compatible with Go 1.24.1)
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <claude@anthropic.com>
This pull request adds support for presets to coder provisioners.
If a template defines presets using a compatible version of the
provider, then this PR will allow those presets to be persisted to the
control plane database for use in workspace creation.
Relates to https://github.com/coder/coder/issues/14062
Previously a `logProvisionerJobLog` helper was added in
https://github.com/coder/coder/pull/6508 to forward logs from the
provisioner at the correct log level, but this was only used for logs
produced in `buildWorkspace`.
This PR uses this helper for forwarding logs produced in
`runTemplateImportParse` and
`runTemplateImportProvisionWithRichParameters` at the correct log level.
This PR introduces a new prometheus metrics for
`workspace_build_timing_seconds`, which specifically reports workspace
build times. To reduce cardinality, this metrics excludes
`workspace_name` and `workspace_owner` that are present on the
`workspace_builds_total` metrics.
Addresses https://github.com/coder/nexus/issues/35.
This PR:
- Adds a `workspace_modules` table to track modules used by the
Terraform provisioner in provisioner jobs.
- Adds a `module_path` column to the `workspace_resources` table,
allowing to identify which module a resource originates from.
- Starts pushing this new information into telemetry.
For the person reviewing this PR, do not fret about the 1,500 new lines
- ~1,000 of them are auto-generated.
* chore: move `/gitauth` to `/externalauth` on the frontend
This actually took a lot more jank than anticipated,
so I wanted to split this up before adding the ability
to embed new providers.
* Rename FE
* Fix em' up
* Fix linting error
* Fix e2e tests
* chore: update helm golden files
* chore: rename `git_auth` to `external_auth` in our schema
We're changing Git auth to be external auth. It will support
any OAuth2 or OIDC provider.
To split up the larger change I want to contribute the schema
changes first, and I'll add the feature itself in another PR.
* Fix names
* Fix outdated view
* Rename some additional places
* Fix sort order
* Fix template versions auth route
* Fix types
* Fix dbauthz
Due to a logical error in CommitQuota, all workspace Stop->Start operations
were being accepted, regardless of the Quota limit. This issue only
appeared after #9201, so this was a minor regression in main for about
3 days. This PR adds a test to make sure this kind of bug doesn't recur.
To make the new test possible, we give the echo provisioner the ability
to simulate responses to specific transitions.
* chore: add /v2 to import module path
go mod requires semantic versioning with versions greater than 1.x
This was a mechanical update by running:
```
go install github.com/marwan-at-work/mod/cmd/mod@latest
mod upgrade
```
Migrate generated files to import /v2
* Fix gen
* Standardize on function to get workspace channel name
There were two, now there is one.
* Add some missing workspace updates
There are some failure cases where we do not set the type as a workspace
build which causes the workspace update to never be published.
* Make build failures warnings
Otherwise the associated test fails due to the logger fataling on
error messages.
- Previously, we were logging all provision response logs at level INFO, regardless of the log level of the log streamed from the provisioner. We now log these at the original level (defaulting to INFO).
- Now logging "provision failed" message at level ERROR WARN and including the error field in the message.
* Add git auth providers schema
* Pipe git auth providers to the schema
* Add git auth providers to the API
* Add gitauth endpoint to query authenticated state
* Add endpoint to query git state
* Use BroadcastChannel to automatically authenticate with Git
* Add error validation for submitting the create workspace form
* Fix panic on template dry-run
* Add tests for the template version Git auth endpoint
* Show error if no gitauth is configured
* Add gitauth to cliui
* Fix unused method receiver
* Fix linting errors
* Fix dbauthz querier test
* Fix make gen
* Add JavaScript test for git auth
* Fix bad error message
* Fix provisionerd test race
See https://github.com/coder/coder/actions/runs/4277960646/jobs/7447232814
* Fix requested changes
* Add comment to CreateWorkspacePageView
This PR adds the prometheus metric coderd_workspace_builds_total.
It measures the total number of workspace builds, along with a number of labels intended to be useful for an operator debugging a failed workspace build trying to discover the scope of the issue.
- rbac: export rbac.Permissions
- dbauthz: move GetDeploymentDAUs, GetTemplateDAUs,
GetTemplateAverageBuildTime from querier.go to system.go
and removes auth checks
- dbauthz: remove AsSystem(), add individual roles for
autostart, provisionerd, add restricted system role for
everything else
Danny Kopping