Cherry-pick of #22461 to `release/2.31`.
Applies the non-browser websocket auth principle from #22226 to
remaining
`codersdk` websocket callsites, replacing cookie-jar session auth with
header-token auth. Fixes `401` failures on deployments with
`--host-prefix-cookie` enabled.
Closes#22461 (cherry-pick)
---------
Co-authored-by: ethan <ethanndickson@gmail.com>
* Adds support for parameter `format=text` in the following API routes:
* `/api/v2/workspaceagents/:id/logs`
* `/api/v2/workspacebuilds/:id/logs`
* `/api/v2/templateversions/:id/logs`
* `/api/v2/templateversions/:id/dry-run/:id/logs`
* Adds links to view raw logs on the following pages:
* Workspace build page
* Template editor page
* Template version page
* Refactors existing log formatting in `cli/logs.go` to live in `codersdk`.
🤖 Generated with Claude Opus 4.5, reviewed by me.
---------
Co-authored-by: Claude <noreply@anthropic.com>
Relates to https://github.com/coder/internal/issues/934
This PR provides a mechanism to filter provisioner jobs according to who
initiated the job.
This will be used to find pending prebuild jobs when prebuilds have
overwhelmed the provisioner job queue. They can then be canceled.
If prebuilds are overwhelming provisioners, the following steps will be
taken:
```bash
# pause prebuild reconciliation to limit provisioner queue pollution:
coder prebuilds pause
# cancel pending provisioner jobs to clear the queue
coder provisioner jobs list --initiator="prebuilds" --status="pending" | jq ... | xargs -n1 -I{} coder provisioner jobs cancel {}
# push a fixed template and wait for the import to complete
coder templates push ... # push a fixed template
# resume prebuild reconciliation
coder prebuilds resume
```
This interface differs somewhat from what was specified in the issue,
but still provides a mechanism that addresses the issue. The original
proposal was made by myself and this simpler implementation makes sense.
I might add a `--search` parameter in a follow-up if there is appetite
for it.
Potential follow ups:
* Support for this usage: `coder provisioner jobs list --search
"initiator:prebuilds status:pending"`
* Adding the same parameters to `coder provisioner jobs cancel` as a
convenience feature so that operators don't have to pipe through `jq`
and `xargs`
## Summary
In this pull request we're adding support in the CLI for prompting the
user for any missing required template variables in the `coder templates
push` command and automatically retrying the template build once a user
has provided any missing variable values.
Closes: https://github.com/coder/coder/issues/19782
### Demo
In the following recording I created a simple template terraform file
that used different variable types (string, number, boolean, and
sensitive) and prompted the user to enter a value for each variable.
<details>
<summary>See example template terraform file</summary>
```tf
...
# Required variables for testing interactive prompting
variable "docker_image" {
description = "Docker image to use for the workspace"
type = string
}
variable "workspace_name" {
description = "Name of the workspace"
type = string
}
variable "cpu_limit" {
description = "CPU limit for the container (number of cores)"
type = number
}
variable "enable_gpu" {
description = "Enable GPU access for the container"
type = bool
}
variable "api_key" {
description = "API key for external services (sensitive)"
type = string
sensitive = true
}
# Optional variable with default
variable "docker_socket" {
default = "/var/run/docker.sock"
description = "Docker socket path"
type = string
}
...
```
</details>
Once the user entered a valid value for each variable, the template
build would be retried.
https://github.com/user-attachments/assets/770cf954-3cbc-4464-925e-2be4e32a97de
<details>
<summary>See output from recording</summary>
```shell
$ ./scripts/coder-dev.sh templates push test-required-params -d examples/templates/test-required-params/
INFO : Overriding codersdk.SessionTokenCookie as we are developing inside a Coder workspace.
/home/coder/coder/build/coder-slim_2.26.0-devel+a68122ca3_linux_amd64
Provisioner tags: <none>
WARN: No .terraform.lock.hcl file found
| When provisioning, Coder will be unable to cache providers without a lockfile and must download them from the internet each time.
| Create one by running terraform init in your template directory.
> Upload "examples/templates/test-required-params"? (yes/no) yes
=== ✔ Queued [0ms]
==> ⧗ Running
==> ⧗ Running
=== ✔ Running [4ms]
==> ⧗ Setting up
=== ✔ Setting up [0ms]
==> ⧗ Parsing template parameters
=== ✔ Parsing template parameters [8ms]
==> ⧗ Cleaning Up
=== ✘ Cleaning Up [4ms]
=== ✘ Cleaning Up [8ms]
Found 5 missing required variables:
- docker_image (string): Docker image to use for the workspace
- workspace_name (string): Name of the workspace
- cpu_limit (number): CPU limit for the container (number of cores)
- enable_gpu (bool): Enable GPU access for the container
- api_key (string): API key for external services (sensitive)
The template requires values for the following variables:
var.docker_image (required)
Description: Docker image to use for the workspace
Type: string
Current value: <empty>
> Enter value: image-name
var.workspace_name (required)
Description: Name of the workspace
Type: string
Current value: <empty>
> Enter value: workspace-name
var.cpu_limit (required)
Description: CPU limit for the container (number of cores)
Type: number
Current value: <empty>
> Enter value: 1
var.enable_gpu (required)
Description: Enable GPU access for the container
Type: bool
Current value: <empty>
? Select value: false
var.api_key (required), sensitive
Description: API key for external services (sensitive)
Type: string
Current value: <empty>
> Enter value: (*redacted*) ******
Retrying template build with provided variables...
=== ✔ Queued [0ms]
==> ⧗ Running
==> ⧗ Running
=== ✔ Running [2ms]
==> ⧗ Setting up
=== ✔ Setting up [0ms]
==> ⧗ Parsing template parameters
=== ✔ Parsing template parameters [7ms]
==> ⧗ Detecting persistent resources
2025-09-25 22:34:14.731Z Terraform 1.13.0
2025-09-25 22:34:15.140Z data.coder_provisioner.me: Refreshing...
2025-09-25 22:34:15.140Z data.coder_workspace.me: Refreshing...
2025-09-25 22:34:15.140Z data.coder_workspace_owner.me: Refreshing...
2025-09-25 22:34:15.141Z data.coder_provisioner.me: Refresh complete after 0s [id=2bd73098-d127-4362-b3a5-628e5bce6998]
2025-09-25 22:34:15.141Z data.coder_workspace_owner.me: Refresh complete after 0s [id=c2006933-4f3e-4c45-9e04-79612c3a5eca]
2025-09-25 22:34:15.141Z data.coder_workspace.me: Refresh complete after 0s [id=36f2dc6f-0bf2-43bd-bc4d-b29768334e02]
2025-09-25 22:34:15.186Z coder_agent.main: Plan to create
2025-09-25 22:34:15.186Z module.code-server[0].coder_app.code-server: Plan to create
2025-09-25 22:34:15.186Z docker_volume.home_volume: Plan to create
2025-09-25 22:34:15.186Z module.code-server[0].coder_script.code-server: Plan to create
2025-09-25 22:34:15.187Z docker_container.workspace[0]: Plan to create
2025-09-25 22:34:15.187Z Plan: 5 to add, 0 to change, 0 to destroy.
=== ✔ Detecting persistent resources [3104ms]
==> ⧗ Detecting ephemeral resources
2025-09-25 22:34:16.033Z Terraform 1.13.0
2025-09-25 22:34:16.428Z data.coder_workspace.me: Refreshing...
2025-09-25 22:34:16.428Z data.coder_provisioner.me: Refreshing...
2025-09-25 22:34:16.429Z data.coder_workspace_owner.me: Refreshing...
2025-09-25 22:34:16.429Z data.coder_provisioner.me: Refresh complete after 0s [id=2d2f7083-88e6-425c-9df3-856a3bf4cc73]
2025-09-25 22:34:16.429Z data.coder_workspace.me: Refresh complete after 0s [id=c723575e-c7d3-43d7-bf54-0e34d0959dc3]
2025-09-25 22:34:16.431Z data.coder_workspace_owner.me: Refresh complete after 0s [id=d43470c2-236e-4ae9-a977-6b53688c2cb1]
2025-09-25 22:34:16.453Z coder_agent.main: Plan to create
2025-09-25 22:34:16.453Z docker_volume.home_volume: Plan to create
2025-09-25 22:34:16.454Z Plan: 2 to add, 0 to change, 0 to destroy.
=== ✔ Detecting ephemeral resources [1278ms]
==> ⧗ Cleaning Up
=== ✔ Cleaning Up [6ms]
┌──────────────────────────────────┐
│ Template Preview │
├──────────────────────────────────┤
│ RESOURCE │
├──────────────────────────────────┤
│ docker_container.workspace │
│ └─ main (linux, amd64) │
├──────────────────────────────────┤
│ docker_volume.home_volume │
└──────────────────────────────────┘
The test-required-params template has been created at Sep 25
22:34:16! Developers can provision a workspace with this template using:
Updated version at Sep 25 22:34:16!
```
</details>
### Changes
- Added a new function to check if the provisioner failed due to a
template missing required variables
- Added a handler function that is called when a provisioner fails due
to the "missing required variables" error. The handler function will:
- Check for provided template variables and identify any missing
variables
- Prompt the user for any missing variables (prompt is adapted based on
the variable type)
- Validate user input for missing variables
- Retry the template build when all variables have been provided by the
user
### Testing
Added tests for the following scenarios:
- Ensure validation based on variable type
- Ensure users are not prompted for variables with a default value
- Ensure variables provided via a variables files (`--variables-file`)
or a variable flag (`--variable`) take precedence over a template
## Summary
In this pull request we're adding support for additional filtering
options to the `provisioners list` CLI command and the
`/provisionerdaemons` API endpoint.
Resolves: https://github.com/coder/coder/issues/18783
### Changes
#### Added CLI Options
- `--show-offline`: When this option is provided, all provisioner
daemons will be returned. This means that when `--show-offline` is not
provided only `idle` and `busy` provisioner daemons will be returned.
- `--status=<list_of_statuses>`: When this option is provided with a
comma-separated list of valid statuses (`idle`, `busy`, or `offline`)
only provisioner daemons that have these statuses will be returned.
- `--max-age=<duration>`: When this option is provided with a valid
duration value (e.g., `24h`, `30s`) only provisioner daemons with a
`last_seen_at` timestamp within the provided max age will be returned.
#### Query Params
- `?offline=true`: Include offline provisioner daemons in the results.
Offline provisioner daemons will be excluded if `?offline=false` or if
offline is not provided.
- `?status=<list_of_statuses>`: Include provisioner daemons with the
specified statuses.
- `?max_age=<duration>`: Include provisioner daemons with a
`last_seen_at` timestamp within the max age duration.
#### Frontend
- Since offline provisioners will not be returned by default anymore
(`--show-offline` has to be provided to see them), a checkbox was added
to the provisioners list page to allow for offline provisioners to be
displayed
- A revamp of the provisioners page will be done in:
https://github.com/coder/coder/issues/17156, this checkbox change was
just added to maintain currently functionality with the backend updates
Current provisioners page (without checkbox)
<img width="1329" height="574" alt="Screenshot 2025-08-20 at 10 51
00 AM"
src="https://github.com/user-attachments/assets/77b73650-0b62-44f0-a77f-acbe5710809f"
/>
Provisioners page with checkbox (unchecked)
<img width="1314" height="626" alt="Screenshot 2025-08-20 at 10 48
40 AM"
src="https://github.com/user-attachments/assets/7ba164ad-6d3f-417b-bd39-338c0161b145"
/>
Provisioner page with checkbox (checked) and URL updated with query
parameters
<img width="1306" height="597" alt="Screenshot 2025-08-20 at 10 50
14 AM"
src="https://github.com/user-attachments/assets/e78d0986-bbf8-491b-9d56-b682973237a0"
/>
### Show Offline vs Offline Status
To list offline provisioner daemons, users can either:
1. Include the `--show-offline` option
OR
2. Include `offline` in the list of values provided to the `--status`
option
This PR sets a constraint of 1MB on the provisioner job logs written to
the database. This is consistent with the constraint we place on
workspace agent logs:
https://github.com/coder/coder/blob/4ac6be6d835dc36c242e35a26b584b784040bf28/coderd/database/dump.sql#L2030
It also adds a message printed to the front end about the provisioner
log overflow, and updates the message printed to the front end when
workspace startup logs exceed the max, as it was causing some customers
to think their startup script had failed to run.
# Description
This PR adds the `worker_name` field to the provisioner jobs endpoint.
To achieve this, the following SQL query was updated:
-
`GetProvisionerJobsByOrganizationAndStatusWithQueuePositionAndProvisioner`
As a result, the `codersdk.ProvisionerJob` type, which represents the
provisioner job API response, was modified to include the new field.
**Notes:**
* As mentioned in
[comment](https://github.com/coder/coder/pull/17877#discussion_r2093218206),
the `GetProvisionerJobsByIDsWithQueuePosition` query was not changed due
to load concerns. This means that for template and template version
endpoints, `worker_id` will still be returned, but `worker_name` will
not.
* Similar to `worker_id`, the `worker_name` is only present once a job
is assigned to a provisioner daemon. For jobs in a pending state (not
yet assigned), neither `worker_id` nor `worker_name` will be returned.
---
# Affected Endpoints
- `/organizations/{organization}/provisionerjobs`
- `/organizations/{organization}/provisionerjobs/{job}`
---
# Testing
- Added new tests verifying that both `worker_id` and `worker_name` are
returned once a provisioner job reaches the **succeeded** state.
- Existing tests covering state transitions and other logic remain
unchanged, as they test different scenarios.
---
# Front-end Changes
Admin provisioner jobs dashboard:
<img width="1088" alt="Screenshot 2025-05-16 at 11 51 33"
src="https://github.com/user-attachments/assets/0e20e360-c615-4497-84b7-693777c5443e"
/>
Fixes: https://github.com/coder/coder/issues/16982
The experimental functions in `golang.org/x/exp/slices` are now
available in the standard library since Go 1.21.
Reference: https://go.dev/doc/go1.21#slices
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
`ServeProvisionerDaemonRequest` has had an ID field for quite a while
now.
This field is only used for telemetry purposes; the actual daemon ID is
created upon insertion in the database. There's no reason to set it, and
it's confusing to do so. Deprecating the field and removing references
to it.
Migrates us to `coder/websocket` v1.8.12 rather than `nhooyr/websocket` on an older version.
Works around https://github.com/coder/websocket/issues/504 by adding an explicit test for `xerrors.Is(err, io.EOF)` where we were previously getting `io.EOF` from the netConn.
fixes#14881
Our handlers for streaming logs don't read from the websocket. We don't allow the client to send us any data, but the websocket library we use requires reading from the websocket to properly handle pings and closing. Not doing so can [can cause the websocket to hang on write](https://github.com/coder/websocket/issues/405), leaking go routines which were noticed in #14881.
This fixes the issue, and in process refactors our log streaming to a encoder/decoder package which provides generic types for sending JSON over websocket.
I'd also like for us to upgrade to the latest https://github.com/coder/websocket but we should also upgrade our tailscale fork before doing so to avoid including two copies of the websocket library.
* Modifies `MatchedProvisioners` response of `codersdk.TemplateVersion`
to be a pointer
* CLI now checks for absence of `*MatchedProvisioners` before showing
warning regarding provisioners
* Extracts logic for warning about provisioners to a function
* Improves test coverage for CLI template push with
`coder_workspace_tags`.
Relates to https://github.com/coder/coder/issues/15087 and
https://github.com/coder/coder/issues/15427
- Extracts provisioner job tags from `coder_workspace_tags` on template
version creation using `provisioner/terraform/tfparse` added in
https://github.com/coder/coder/pull/15236
- Drops a WARN log in coderd if no matching provisioners found.
- Also drops a warning message in the CLI if no provisioners are found.
- To support both CLI and UI warnings, added a
`codersdk.MatchedProvisioners` struct to the `TemplateVersion` response
containing details of how many provisioners were around at the time of
the insert.
Co-authored-by: Mathias Fredriksson <mafredri@gmail.com>
This PR is the first step aiming to resolve#15126 -
Creating a new endpoint to return the details associated to a
provisioner key.
This is an authenticated endpoints aiming to be used by the provisioner
daemons - using the provisioner key as authentication method.
This endpoint is not ment to be used with PSK or User Sessions.
Closes#14716Closes#14717
Adds a new user-scoped tailnet API endpoint (`api/v2/tailnet`) with a new RPC stream for receiving updates on workspaces owned by a specific user, as defined in #14716.
When a stream is started, the `WorkspaceUpdatesProvider` will begin listening on the user-scoped pubsub events implemented in #14964. When a relevant event type is seen (such as a workspace state transition), the provider will query the DB for all the workspaces (and agents) owned by the user. This gets compared against the result of the previous query to produce a set of workspace updates.
Workspace updates can be requested for any user ID, however only workspaces the authorised user is permitted to `ActionRead` will have their updates streamed.
Opening a tunnel to an agent requires that the user can perform `ActionSSH` against the workspace containing it.
I'm seeing flakes like
```
provisionerkeys_test.go:68: 2024-09-30 05:58:44.686: cmd: matched newline = "CREATED AT NAME TAGS "
provisionerkeys_test.go:72: 2024-09-30 05:58:44.686: cmd: matched newline = "2024-09-30T05:58:44Z dont-test-me my=way foo=bar "
provisionerkeys_test.go:74:
Error Trace: /Users/runner/work/coder/coder/enterprise/cli/provisionerkeys_test.go:74
Error: "2024-09-30T05:58:44Z dont-test-me my=way foo=bar " does not contain "foo=bar my=way"
Test: TestProvisionerKeys/CRUD
```
e.g.
https://github.com/coder/coder/actions/runs/11100237276/job/30835714478?pr=14855
Since the tags are a map, we weren't outputting them in a consistent
order on the CLI, leading to flakes.
This sorts the tags by key when converting to a string, for a
consistent, canonical output.
What this changes:
- Unhides the `--key` flag on provisioner start
- Deprecates and hides `provisionerd` command group in favor of
`provisioner(s)`
- Removes org id from `coder provisioner keys list`
* assert provisioner daemon version and api_version in unit tests
* add build info in HTTP header, extract codersdk.BuildVersionHeader
* add api_version to codersdk.ProvisionerDaemon
* testutil.MustString -> testutil.MustRandString
Part of #10532
DRPC transport over yamux and in-mem pipes was previously only used on the provisioner APIs, but now will also be used in tailnet. Moved to subpackage of codersdk to avoid import loops.
- Adds a --name argument to provisionerd start
- Plumbs through name to integrated and external provisioners
- Defaults to hostname if not specified for external, hostname-N for integrated
- Adds cliutil.Hostname
This change removes one use of `coderd/database` from the slim binary
and more correctly uses codersdk instead of database or provisionerd
packages.
No size change (yet).
Ref: #9380
* chore: add /v2 to import module path
go mod requires semantic versioning with versions greater than 1.x
This was a mechanical update by running:
```
go install github.com/marwan-at-work/mod/cmd/mod@latest
mod upgrade
```
Migrate generated files to import /v2
* Fix gen
Jakub Domeracki