**This is just the protobuf changes for the PR https://github.com/coder/coder/pull/21398**
Moved `UploadFileRequest` from `provisionerd.proto` -> `provisioner.proto`.
Renamed to `FileUpload` because it is now bi-directional.
This **is backwards compatible**. I tested it to confirm the payloads are identical. Types were just renamed and moved around.
```golang
func TestTypeUpgrade(t *testing.T) {
t.Parallel()
x := &proto2.UploadFileRequest{
Type: &proto2.UploadFileRequest_ChunkPiece{
ChunkPiece: &proto.ChunkPiece{
Data: []byte("Hello World!"),
FullDataHash: []byte("Foobar"),
PieceIndex: 42,
},
},
}
data, err := protobuf.Marshal(x)
require.NoError(t, err)
// Exactly the same output
// EhgKDEhlbGxvIFdvcmxkIRIGRm9vYmFyGCo= on `main`
// EhgKDEhlbGxvIFdvcmxkIRIGRm9vYmFyGCo= on this branch
fmt.Println(base64.StdEncoding.EncodeToString(data))
}
```
# What this does
This allows provisioner daemons to download files from `coderd`'s `files` table. This is used to send over cached module files and prevent the need of downloading these modules on each workspace build.
Adds some extra meta data sent to provisioners. Also adds a field
`reuse_terraform_workspace` to tell the provisioner whether or not to
use the caching experiment.
This pull request introduces support for external workspace management, allowing users to register and manage workspaces that are provisioned and managed outside of the Coder.
Depends on: https://github.com/coder/terraform-provider-coder/pull/424
* GET /api/v2/init-script - Gets the agent initialization script
* By default, it returns a script for Linux (amd64), but with query parameters (os and arch) you can get the init script for different platforms
* GET /api/v2/workspaces/{workspace}/external-agent/{agent}/credentials - Gets credentials for an external agent **(enterprise)**
* Updated queries to filter workspaces/templates by the has_external_agent field
This PR implements protobuf streaming to handle large module files by:
1. **Streaming large payloads**: When module files exceed the 4MB limit,
they're streamed in chunks using a new UploadFile RPC method
2. **Database storage**: Streamed files are stored in the database and
referenced by hash for deduplication
3. **Backward compatibility**: Small module files continue using the
existing direct payload method
Closes https://github.com/coder/internal/issues/369
We can't know whether a replacement (i.e. drift of terraform state
leading to a resource needing to be deleted/recreated) will take place
apriori; we can only detect it at `plan` time, because the provider
decides whether a resource must be replaced and it cannot be inferred
through static analysis of the template.
**This is likely to be the most common gotcha with using prebuilds,
since it requires a slight template modification to use prebuilds
effectively**, so let's head this off before it's an issue for
customers.
Drift details will now be logged in the workspace build logs:
Plus a notification will be sent to template admins when this situation
arises:
A new metric - `coderd_prebuilt_workspaces_resource_replacements_total`
- will also increment each time a workspace encounters replacements.
We only track _that_ a resource replacement occurred, not how many. Just
one is enough to ruin a prebuild, but we can't know apriori which
replacement would cause this.
For example, say we have 2 replacements: a `docker_container` and a
`null_resource`; we don't know which one might
cause an issue (or indeed if either would), so we just track the
replacement.
---------
Signed-off-by: Danny Kopping <dannykopping@gmail.com>
This pull request adds support for presets to coder provisioners.
If a template defines presets using a compatible version of the
provider, then this PR will allow those presets to be persisted to the
control plane database for use in workspace creation.
Replace Depot build action with Nix for Nix dogfood image builds
The dogfood Nix image is now built using Nix's native container tooling instead of Depot. This change:
- Adds Nix setup steps to the GitHub Actions workflow
- Removes the Dockerfile.nix in favor of a Nix-native container build
- Updates the flake.nix to support building Docker images
- Introduces a hash file to track Nix-related changes
- Updates the vendorHash for Go dependencies
Change-Id: I4e011fe3a19d9a1375fbfd5223c910e59d66a5d9
Signed-off-by: Thomas Kosiewski <tk@coder.com>
Addresses https://github.com/coder/nexus/issues/35.
This PR:
- Adds a `workspace_modules` table to track modules used by the
Terraform provisioner in provisioner jobs.
- Adds a `module_path` column to the `workspace_resources` table,
allowing to identify which module a resource originates from.
- Starts pushing this new information into telemetry.
For the person reviewing this PR, do not fret about the 1,500 new lines
- ~1,000 of them are auto-generated.
* chore: rename `git_auth` to `external_auth` in our schema
We're changing Git auth to be external auth. It will support
any OAuth2 or OIDC provider.
To split up the larger change I want to contribute the schema
changes first, and I'll add the feature itself in another PR.
* Fix names
* Fix outdated view
* Rename some additional places
* Fix sort order
* Fix template versions auth route
* Fix types
* Fix dbauthz
* chore: revert nix-related CI changes
- Reverts using nix to run CI-dependencies.
- Running 'make gen' in a dogfood workspace resulted in inconsistent
results for protobuf-related files making it difficult to pass CI.
This PR imports the minimum changes necessary to make CI compatible
with dogfood.
* chore: add /v2 to import module path
go mod requires semantic versioning with versions greater than 1.x
This was a mechanical update by running:
```
go install github.com/marwan-at-work/mod/cmd/mod@latest
mod upgrade
```
Migrate generated files to import /v2
* Fix gen
* Add Docker image
* Try building the container in CI
* Uncomment testing lines
* Trim image step
* Hit 'em with a Docker load
* Oopsie!
* Add a tag and push!
* Fix image name 🤦
* Fix sudo
* Fix target name
* Build and push
* Fix login to docker hub
* Revert to Docker push
* Fix PATH
* Fix SHA
* Fix ca certs
* Fix ca certs
* Fix coping files in
* Fix docker init
* Fix Docker group and init
* Add comments to our Nix
* Fix build stage
* Add some more comments
* Remove old dogfood image files
* Tag and push with branch name
* Fix tag passing
* Fix tag passing
* Remove old pull triggers
* Convert gen to use Nix
* Add protobuf to the flake
* Add prettier to the dev shell
* Swap to the faster Nix cache action
* Add the correct yq
* Fix gen
* Add make to the flake
* Update extensions
* Add git auth providers schema
* Pipe git auth providers to the schema
* Add git auth providers to the API
* Add gitauth endpoint to query authenticated state
* Add endpoint to query git state
* Use BroadcastChannel to automatically authenticate with Git
* Add error validation for submitting the create workspace form
* Fix panic on template dry-run
* Add tests for the template version Git auth endpoint
* Show error if no gitauth is configured
* Add gitauth to cliui
* Fix unused method receiver
* Fix linting errors
* Fix dbauthz querier test
* Fix make gen
* Add JavaScript test for git auth
* Fix bad error message
* Fix provisionerd test race
See https://github.com/coder/coder/actions/runs/4277960646/jobs/7447232814
* Fix requested changes
* Add comment to CreateWorkspacePageView
Customer feedback indicated projects was a confusing name.
After querying the team internally, it seemed unanimous
that it is indeed a confusing name.
Here's for a lil less confusion @ashmeer7 🥂
* feat: Add stage to build logs
This adds a stage property to logs, and refactors the job logs
cliui.
It also adds tests to the cliui for build logs!
* Fix comments
* ci: Update DataDog GitHub branch to fallback to GITHUB_REF
This was detecting branches, but not our "main" branch before.
Hopefully this fixes it!
* Add basic Terraform Provider
* Rename post files to upload
* Add tests for resources
* Skip instance identity test
* Add tests for ensuring agent get's passed through properly
* Fix linting errors
* Add echo path
* Fix agent authentication
* fix: Convert all jobs to use a common resource and agent type
This enables a consistent API for project import and provisioned resources.
* Add "coder_workspace" data source
* feat: Remove magical parameters from being injected
This is a much cleaner abstraction. Explicitly declaring the user
parameters for each provisioner makes for significantly simpler
testing.
* feat: Add graceful exits to provisionerd
Terraform (or other provisioners) may need to cleanup state, or
cancel actions before exit. This adds the ability to gracefully
exit provisionerd.
* Fix cancel error check
* ci: Update DataDog GitHub branch to fallback to GITHUB_REF
This was detecting branches, but not our "main" branch before.
Hopefully this fixes it!
* Add basic Terraform Provider
* Rename post files to upload
* Add tests for resources
* Skip instance identity test
* Add tests for ensuring agent get's passed through properly
* Fix linting errors
* Add echo path
* Fix agent authentication
* fix: Convert all jobs to use a common resource and agent type
This enables a consistent API for project import and provisioned resources.
* Add "coder_workspace" data source
* feat: Remove magical parameters from being injected
This is a much cleaner abstraction. Explicitly declaring the user
parameters for each provisioner makes for significantly simpler
testing.
Steven Masley