Fixes all our Go file imports to match the preferred spec that we've _mostly_ been using. For example:
```
import (
"context"
"time"
"github.com/prometheus/client_golang/prometheus"
"golang.org/x/xerrors"
"gopkg.in/natefinch/lumberjack.v2"
"cdr.dev/slog/v3"
"github.com/coder/coder/v2/codersdk/agentsdk"
"github.com/coder/serpent"
)
```
3 groups: standard library, 3rd partly libs, Coder libs.
This PR makes the change across the codebase. The PR in the stack above modifies our formatting to maintain this state of affairs, and is a separate PR so it's possible to review that one in detail.
This PR adds a command to view the provisioner and agent logs for a
given workspace.
Note: I did investigate using the existing `cliui` methods to tail the
logs but they are tailored to a very specific use-case.
Other changes:
- Adds `Agents` to `dbfake.WorkspaceResponse`
- Adds methods to generate provisioner and agent logs in `dbgen`
---------
Co-authored-by: Steven Masley <Emyrk@users.noreply.github.com>
Upgrades to slog v3 which includes a small, but backward incompatible API change to the acceptible call arguments when logging. This change allows us to verify via compile time type checking that arguments are correct and won't cause a panic, as was possible in slog v1, which this replaces (v2 was tagged but never used in coder/coder).
It also updates dependencies that also use slog and were updated.
I've left the `aibridge` dependency as a commit SHA, under the assumption that the team there (cc @pawbana @dannykopping ) will tag and update the dependency soon and on their own schedule.
Other dependencies, I pushed new tags.
Fixes https://github.com/coder/internal/issues/272
This test periodically fails due to the healthcheck timing out.
The problem is compounded due to the fact that we stand up a new
coderdtest instance for each test.
This PR does the following:
* Updates the subtests to share a single `coderdtest` instance.
* Hits the `/debug/health` endpoint before completing the setup phase so
that the result is cached.
This will not completely remove the issue, as the healthcheck could
still fail due to test-infrastructure-related issues. In this case we
may decide to add a retry in this 'seed' function.
Because this affects more than just the template insights
page (specifically it also affects the deployment stats endpoint which
is shown on bottom bar and Prometheus), the group is being renamed
generically to just "stats collection". In the future if we need to
affect the other stats we can put those options here.
Then, because this change only affects a portion of stats, specifically
usage stats like connection and application time, bytes sent, etc, add a
new sub-group called "usage stats".
Then finally add back the "enable" flag. This also gives us a place to
one day place an "anonymize" flag if we need to go that route.
Add agent forwarding of boundary audit logs from workspaces to coderd
via agent API, and re-emission of boundary logs to coderd stderr. This
change adds a server to the workspace agent that always listens on a
unix socket for boundary to connect and send audit logs.
coderd log format example:
```
[API] 2025-12-23 18:31:46.755 [info] coderd.agentrpc: boundary_request owner=.. workspace_name=.. agent_name=.. decision=.. workspace_id=.. http_method=.. http_url=.. event_time=.. request_id=..
```
Corresponding boundary PR: https://github.com/coder/boundary/pull/124
RFC:
https://www.notion.so/coderhq/Agent-Boundary-Logs-2afd579be59280f29629fc9823ac41bahttps://github.com/coder/coder/issues/21280
## Description
Adds the core AI Bridge MITM proxy daemon. This proxy intercepts HTTPS traffic, decrypts it using a configured CA certificate, and forwards requests to AIBridge for processing.
## Changes
* Added `aibridgeproxyd` package with the core proxy server implementation
* Added configuration options: `CODER_AIBRIDGE_PROXY_ENABLED`, `CODER_AIBRIDGE_PROXY_LISTEN_ADDR`, `CODER_AIBRIDGE_PROXY_CERT_FILE`, `CODER_AIBRIDGE_PROXY_KEY_FILE`
* Added tests for server initialization and MITM functionality
Closes https://github.com/coder/internal/issues/1180
A simple typo fix to the help text
stidin > stdin
```
➜ coder git:(org_role_fix) ✗ coder organizations roles create -h
coder v2.29.1+59cdd7e
USAGE:
coder organizations roles create [flags] <role_name>
Create a new organization custom role
- Run with an input.json file:
$ coder organization -O <organization_name> roles create --stidin < role.json
```
Related to
[`internal#1139`](https://github.com/coder/internal/issues/1139)
This implements some prometheus metrics for records being removed from
the database. Currently we're tracking the following fields being
removed from the DB by this. They're viewable in the
`/api/v2/debug/metrics` endpoint.
* `expired_api_keys`
* `aibridge_records`
* `connection_logs`
* `duration`
```
# HELP coderd_dbpurge_iteration_duration_seconds Duration of each dbpurge iteration in seconds.
# TYPE coderd_dbpurge_iteration_duration_seconds histogram
coderd_dbpurge_iteration_duration_seconds_bucket{success="true",le="1"} 1
coderd_dbpurge_iteration_duration_seconds_bucket{success="true",le="5"} 1
coderd_dbpurge_iteration_duration_seconds_bucket{success="true",le="10"} 1
coderd_dbpurge_iteration_duration_seconds_bucket{success="true",le="30"} 1
coderd_dbpurge_iteration_duration_seconds_bucket{success="true",le="60"} 1
coderd_dbpurge_iteration_duration_seconds_bucket{success="true",le="300"} 1
coderd_dbpurge_iteration_duration_seconds_bucket{success="true",le="600"} 1
coderd_dbpurge_iteration_duration_seconds_bucket{success="true",le="+Inf"} 1
coderd_dbpurge_iteration_duration_seconds_sum{success="true"} 0.014787814
coderd_dbpurge_iteration_duration_seconds_count{success="true"} 1
# HELP coderd_dbpurge_records_purged_total Total number of records purged by type.
# TYPE coderd_dbpurge_records_purged_total counter
coderd_dbpurge_records_purged_total{record_type="aibridge_records"} 0
coderd_dbpurge_records_purged_total{record_type="audit_logs"} 0
coderd_dbpurge_records_purged_total{record_type="connection_logs"} 0
coderd_dbpurge_records_purged_total{record_type="expired_api_keys"} 0
coderd_dbpurge_records_purged_total{record_type="workspace_agent_logs"} 0
```
| Position | Pull-request |
| -------- | ------------ |
| ✅ | [feat: add prometheus observability metrics for
`dbpurge`](https://github.com/coder/coder/pull/21074) |
| | [feat: add rbac specificity for
`dbpurge`](https://github.com/coder/coder/pull/21088) |
While scale testing, I noticed that our load generators send basically
all requests to a single Coderd instance.
e.g.
This is because our scale test commands create all `Runner`s using the
same codersdk Client, which means they share an underlying HTTP client.
With HTTP/2 a single TCP session can multiplex many different HTTP
requests (including websockets). So, it creates a single TCP connection
to a single coderd, and then sends all the requests down the one TCP
connections.
This PR modifies the `exp scaletest` load generator commands to create
an independent HTTP client per `Runner`. This means that each runner
will create its own TCP connection. This should help spread the load and
make a more realistic test, because in a real deployment, scaled out
load will be coming over different TCP connections.
Adds `--max-failures` flag to `coder exp scaletest create-workspaces` so that we can tolerate a few failures without failing the command.
When running our scale test infra, we create Kubernetes Jobs to create the initial cluster workspaces, then we have load-generation jobs that depend on them. At high scale, it's kind of expected that some of the requests will fail: even with 99.9% success, you still expect one failure per 1000. It's useful to be able to carry on with the scale test anyway and proceed to traffic generation.
**Breaking Change:** Existing oauth apps might now use PKCE. If an
unknown IdP type was being used, and it does not support PKCE, it will
break.
To fix, set the PKCE methods on the external auth to `none`
```
export CODER_EXTERNAL_AUTH_1_PKCE_METHODS=none
```
Provisioner steps broken into smaller granular actions.
Changes:
- `ExtractArchive` moved to `init` request (was in `configure`)
- Writing `tfstate` moved to `plan` (was in `configure`)
- Moved most plan/apply outputs to `GraphComplete`
This change ensures keyring tests that utilize the real OS keyring use
credentials that are isolated by process ID so that parallel test processes
do not access the same credentials.
https://github.com/coder/internal/issues/1192
Closes#20399
To summarize the original commit messages:
- Do not log stats to the database.
- Return errors on the insight endpoints.
- Update the frontend to show those errors.
- Also fixes an issue with getting the user status count via codersdk,
since I added a test to ensure it was not disabled by this flag and it
was sending the wrong payload.
Previously the GetTemplateVersionVariables query did not sort output,
relying on PostgreSQL on-disk ordering which is undeterministic.
Variables are now sorted by name because there is no alternative for
ordering.
Tests were adjusted to accommodate the new ordering, previously they
relied on data being written to disk in insert order.
Since the failing test logs are gone, we can only guess at what went
wrong. Given our parallel test-suite, and that tests typically run slow
on Windows, it seems reasonable that the context timed out due to a
single context being responsbile for setup and two command executions.
This change fixes the issue by updating the context usage, if this flake
ever resurfaces, we can re-investigate.
Fixescoder/internal#770
## Summary
This adds configurable overload protection to the AI Bridge daemon to
prevent the server from being overwhelmed during periods of high load.
Partially addresses coder/internal#1153 (rate limits and concurrency
control; circuit breakers are deferred to a follow-up).
## New Configuration Options
| Option | Environment Variable | Description | Default |
|--------|---------------------|-------------|---------|
| `--aibridge-max-concurrency` | `CODER_AIBRIDGE_MAX_CONCURRENCY` |
Maximum number of concurrent AI Bridge requests. Set to 0 to disable
(unlimited). | `0` |
| `--aibridge-rate-limit` | `CODER_AIBRIDGE_RATE_LIMIT` | Maximum number
of AI Bridge requests per second. Set to 0 to disable rate limiting. |
`0` |
## Behavior
When limits are exceeded:
- **Concurrency limit**: Returns HTTP `503 Service Unavailable` with
message "AI Bridge is currently at capacity. Please try again later."
- **Rate limit**: Returns HTTP `429 Too Many Requests` with
`Retry-After` header.
Both protections are optional and disabled by default (0 values).
## Implementation
The overload protection is implemented as reusable middleware in
`coderd/httpmw/ratelimit.go`:
1. **`RateLimitByAuthToken`**: Per-user rate limiting that uses
`APITokenFromRequest` to extract the authentication token, with fallback
to `X-Api-Key` header for AI provider compatibility (e.g., Anthropic).
Falls back to IP-based rate limiting if no token is present. Includes
`Retry-After` header for backpressure signaling.
2. **`ConcurrencyLimit`**: Uses an atomic counter to track in-flight
requests and reject when at capacity.
The middleware is applied in `enterprise/coderd/aibridge.go` via
`r.Group` in the following order:
1. Concurrency check (faster rejection for load shedding)
2. Rate limit check
**Note**: Rate limiting currently applies to all AI Bridge requests,
including pass-through requests. Ideally only actual interceptions
should count, but this would require changes in the aibridge library.
## Testing
Added comprehensive tests for:
- Rate limiting by auth token (Bearer token, X-Api-Key, no token
fallback to IP)
- Different tokens not rate limited against each other
- Disabled when limit is zero
- Retry-After header is set on 429 responses
- Concurrency limiting (allows within limit, rejects over limit,
disabled when zero)
Adds `--disable-workspace-sharing` option.
Workspace sharing is disabled by not including user and group ACLs in
the workspace RBAC object, which prevents ACL-based authz.
Closes https://github.com/coder/internal/issues/1072
The commit also adds saving of workspace user/group ACLs in the test DB
data generator.
Verifies that streamLogs properly returns ctx.Err() when the context is
cancelled while waiting for logs. This covers the case where a user
interrupts an SSH connection (e.g., Ctrl+C) during startup script
execution.
Refs #21104
When users pass --wait=no or set CODER_SSH_WAIT=no, startup logs are no
longer dumped to stderr. The stage indicator is still shown, just not
the log content.
Fixes#13580
The Agent function had complex nested control flow and cross-case state sharing
via the showStartupLogs flag. This made the code hard to follow and maintain.
This change extract an agentWaiter struct with self-contained methods:
- wait: main state machine loop
- waitForConnection: handles Connecting/Timeout states
- handleConnected: handles Connected state and startup scripts
- streamLogs: handles log streaming/fetching
- waitForReconnection: handles Disconnected state
- pollWhile: helper to consolidate polling loops
Each handler is now self-contained with no cross-method state sharing and the
showStartupLogs flag is replaced by return values and the waitedForConnection
tracking variable.
Closes https://github.com/coder/internal/issues/1173,
https://github.com/coder/internal/issues/1174
Currently these two tests are flaky because the contexts were created
before a potentially long-running process. By the time the context was
actually used, it may have timed out - leading to confusion.
Additionally, the `ExpectMatch` calls were not using the test context -
but rather a background context. I've marked that func as deprecated
because we should always tie these to the test context.
Special thanks to @mafredri for the brain probe 🧠
---------
Signed-off-by: Danny Kopping <danny@coder.com>
This changes makes it so that we output the empty string for Format
when there is no data. It turns out there are many places in the code
where we have such handling, but in a way that would break the JSON
formatter (since we'd output nothing on stdout or text rather than
`[]`/`null`).
Replace hardcoded 7-day retention for workspace agent logs with
configurable retention from deployment settings. Defaults to 7d to
preserve existing behavior.
Depends on #21038
Updates #20743
Add `RetentionConfig` with server flags for configuring data retention:
- `--audit-logs-retention`: retention for audit log entries
- `--connection-logs-retention`: retention for connection logs
- `--api-keys-retention`: retention for expired API keys (default 7d)
Updates #20743
## Description
Fixes the prebuilds scaletest command where the prometheus server was
being shut down before waiting for metrics to be scraped.
The issue was the defer order - since defers execute in LIFO (last-in,
first-out) order:
**Before (broken):**
1. Register tracing defer (includes wait for prometheus scrape)
2. Register prometheus server defer
Execution order: prometheus closes first, then wait happens (server
already gone!)
**After (fixed):**
1. Register prometheus server defer
2. Register tracing defer (includes wait for prometheus scrape)
Execution order: wait happens first (server still up), then prometheus
closes.
This matches the pattern used in other scaletest commands.
## Impact
The `coderd_scaletest_prebuild_deletion_jobs_completed` metric (and
potentially others) was always showing 0 because the server shut down
before Prometheus could scrape the final values.
_This PR was generated by [`mux`](https://github.com/coder/mux) and
reviewed by a human._
> [!CAUTION]
> In whichever release this lands, we've removed the ability to provide
keys via a YAML file (specifically on `openai_key`, `anthropic_key`,
`bedrock_access_key` and finally `bedrock_access_key_secret`). This will
need to be described in the release notes as to not break peoples AI
Bridge integrations upgrading from older versions.
This pull-request ensures that we can see the overview of the settings
of the `AI Bridge` feature within the `/deployment/observability` route.
This set of options only render when the `aibridge` feature flag is
enabled.
### Preview
closes: https://github.com/coder/coder/issues/10352
closes: https://github.com/coder/internal/issues/1094
closes: https://github.com/coder/internal/issues/1095
In this pull request, we enable a new set of experimental cli commands
grouped under `coder exp sync`.
These commands allow any process acting within a coder workspace to
inform the coder agent of its requirements and execution progress. The
coder agent will then relay this information to other processes that
have subscribed.
These commands are:
```
# Check if this feature is enabled in your environment
coder exp sync ping
# express that your unit depends on another
coder exp sync want <unit> <dependency_unit>
# express that your unit intends to start a portion of the script that requires
# other units to have completed first. This command blocks until all dependencies have been met
coder exp sync start <unit>
# express that your unit has completes its work, allowing dependent units to begin their execution
coder exp sync complete <unit>
```
Example:
In order to automatically run claude code in a new workspace, it must
first have a git repository cloned. The scripts responsible for cloning
the repository and for running claude code would coordinate in the
following way:
```bash
# Script A: Claude code
# Inform the agent that the claude script wants the git script.
# That is, the git script must have completed before the claude script can begin its execution
coder exp sync want claude git
# Inform the agent that we would now like to begin execution of claude.
# This command will block until the git script (and any other defined dependencies)
# have completed
coder exp sync start claude
# Now we run claude code and any other commands we need
claude ...
# Once our script has completed, we inform the agent, so that any scripts that depend on this one
# may begin their execution
coder exp sync complete claude
```
```bash
# Script B: Git
# Because the git script does not have any dependencies, we can simply inform the agent that we
# intend to start
coder exp sync start git
git clone ssh://git@github.com/coder/coder
# Once the repository have been cloned, we inform the agent that this script is complete, so that
# scripts that depend on it may begin their execution.
coder exp sync complete git
```
Notes:
* Unit names (ie. `claude` and `git`) given as input to the sync
commands are arbitrary strings. You do not have to conform to specific
identifiers. We recommend naming your scripts descriptively, but
succinctly.
* Scripts unit names should be well documented. Other scripts will need
to know the names you've chosen in order to depend on yours. Therefore,
you
---------
Co-authored-by: Mathias Fredriksson <mafredri@gmail.com>
This fixes a regression that caused the VS code extension to be unable
to authenticate after making keyring usage on by default. This is
because the VS code extension assumes the CLI will always use the
session token stored on disk, specifically in the directory specified by
--global-config.
This fix makes keyring usage enabled when the --global-config directory
is not set. This is a bit wonky but necessary to allow the extension to
continue working without modification and without backwards compat
concerns. In the future we should modify these extensions to either
access the credential in the keyring (like Coder Desktop) or some other
approach that doesn't rely on the session token being stored on disk.
Tests:
`coder login dev.coder.com` -> token stored in keyring
`coder login --global-config=/tmp/ dev.coder.com` -> token stored in
`/tmp/session`
Make keyring usage for session token storage on by default for supported
platforms (Windows and macOS), with the ability to opt-out via
--use-keyring=false.
This change will be a breaking change for any users depending on the
session token being stored on disk, though users can restore file usage
via the flag above.
This change will also require CLI users to authenticate after updating.
## Overview
This change promotes the tasks CLI commands from `coder exp task` to
`coder task`, marking them as generally available (GA).
## Migration
Users will need to update their scripts from:
```shell
coder exp task create "my task"
```
To:
```shell
coder task create "my task"
```
---
🤖 This change was written by Claude Sonnet 4.5 Thinking using [mux](https://github.com/coder/mux) and reviewed by a human 🏄🏻♂️
## Problem
Tasks currently only expose a machine-friendly name field (e.g.
`task-python-debug-a1b2`), but this value is primarily an identifier
rather than a clean, descriptive label. We need a separate
display-friendly name for use in the UI.
This PR introduces a new `display_name` field and updates the task-name
generation flow. The Claude system prompt was updated to return valid
JSON with both `name` and `display_name`. The name generation logic
follows a fallback chain (Anthropic > prompt sanitization > random
fallback). To make task names more closely resemble their display names,
the legacy `task-` prefix has been removed. For context, PR
https://github.com/coder/coder/pull/20834 introduced a small Task icon
to the workspace list to help identify workspaces associated to tasks.
## Changes
- Database migration: Added `display_name` column to tasks table
- Updated system prompt to generate both task name and display name as
valid JSON
- Task name generation now follows a fallback chain: Anthropic > prompt
sanitization > random fallback
- Removed `task-` prefix from task names to allow more descriptive names
- Note: PR https://github.com/coder/coder/pull/20834 adds a Task icon to
workspaces in the workspace list to distinguish task-created workspaces
**Note:** UI changes will be addressed in a follow-up PR
Related to: https://github.com/coder/coder/issues/20801
This change updates how SMTP notifications are polled during scale
tests.
Before, each of the ~2,000 pollers created its own http.Client, which
opened thousands of short-lived TCP connections.
Under heavy load, this ran out of available network ports and caused
errors like `connect: cannot assign requested address`
Now, all pollers share one HTTP connection pool. This prevents port
exhaustion and makes polling faster and more stable.
If a network error happens, the poller will now retry instead of
stopping, so tests keep running until all notifications are received.
The `SMTPRequestTimeout` is now applied per request using a context,
instead of being set on the `http.Client`.
Retries were previously added when starting embedded postgres to
mitigate port allocation conflicts (we can't use an ephemeral port for
tests). Retries alone seemingly did not fix the test flakes. A new
failure mode appeared on the retries: timing out connecting to the
database.
When a port discovery error occurrs, embedded-postgres does not create
the database. If the data directory exists on the next attempt,
embedded-postgres will assume the database has already been created.
This seems to cause the timeout error. Wipe all state between retries to
ensure attempts execute the same logic that creates the database.
[#658](https://github.com/coder/internal/issues/658)
Experiments passed to provisioners to determine behavior. This adds
`--experiments` flag to provisioner daemons. Prior to this, provisioners
had no method to turn on/off experiments.
Adds some extra meta data sent to provisioners. Also adds a field
`reuse_terraform_workspace` to tell the provisioner whether or not to
use the caching experiment.
Spike Curtis
