Replace the env-based `BuildProviders` with a DB-backed loader. The database is now the single source of truth for runtime provider configuration; env config arrives via `SeedAIProvidersFromEnv` (run at boot) and `BuildProviders` reads it back as `aibridge.Provider` instances. `cli/server.go` and `enterprise/cli/server.go` both call the same path, so aibridged and aibridgeproxyd see the same provider set.
Per-provider `DumpDir` is replaced by a top-level `CODER_AI_GATEWAY_DUMP_DIR` base; each provider's effective dump path is `<base>/<provider name>`.
We weren't stripping the API base (`/api/v2/aibridge`), leading to 404s
when using the in-memory transport.
Signed-off-by: Danny Kopping <danny@coder.com>
Since AI Gateway is now enabled by default, and if the AI Gateway Proxy is enabled too it's possible the server can start without any configured providers. This would previously block startup, which is unacceptable.
In an upstack PR we will handle reloading the providers at runtime, so the server needs to be able to start up even if it can't handle any proxy requests to AI Gateway.
This change was necessitated because if there are providers configured in the environment they need to be seeded _before_ the proxy starts.
In order to allow Coder Agents to use AI Gateway in OSS, we need to rehome the `aibridged`\-related code into the AGPL path.
The HTTP API is only registered under enterprise so will still require the AI Governance Add-on to be present in order to use it, whereas Coder Agents uses an in-memory pipe to the same handlers.
`CODER_AI_GATEWAY_ENABLED` / `CODER_AIBRIDGE_ENABLED` is now being defaulted to `true` now that it will be used by Coder Agents.
If you previously had this value disabled explicitly, that value will persist.
_Disclaimer: implemented by a Coder Agent using Claude Opus 4.7_
Part of the implementation of [RFC: Common AI Provider Configs](https://www.notion.so/coderhq/RFC-Common-AI-Provider-Configs-34bd579be59280ed958feffb82024797) (AIGOV-201).
## Note
This change can cause a previously working installation to fail to start should a conflict exist between the providers configured in the environment & those now migrated to the database.
I'll raise a PR upstack to document this process and workarounds should a startup fail.
## What this PR does
Reconciles environment-derived AI provider configuration with the `ai_providers` table at server startup. The seed runs **before** the aibridged daemon is initialized, so the runtime always reads providers from the database; the legacy `CODER_AIBRIDGE_*` environment variables become a one-shot migration source.
### Behavior
- Concurrent server starts are serialized through a Postgres advisory lock (`LockIDAIProvidersEnvSeed`).
- Missing rows are inserted with an audit entry attributed to the system actor.
- Existing rows whose canonical hash matches the env-derived hash are left alone (the common no-op restart path).
- Existing rows whose canonical hash does **not** match cause server startup to fail with a descriptive error so the operator can explicitly resolve the conflict in either env or DB.
- Soft-deleted rows are NOT resurrected from env; an explicit operator deletion is sticky across restarts.
- Indexed providers whose name conflicts with a legacy env var fail startup with a clear remediation message.
- Unknown provider types (e.g. `copilot`, until the DB enum is widened) are skipped with a log entry rather than failing startup.
### Canonical hashing
The `canonicalAIProvider` shape captures exactly the fields that determine runtime behavior — `type`, `base_url`, and the Bedrock subset of settings (access key, access key secret, region, model, small fast model) — and is hashed with SHA-256. The hash is **computed on demand from the row + env**, never persisted, so the database does not need a new column for it. API keys live in the separate `ai_provider_keys` table and are intentionally excluded from the hash so operators can rotate keys via the API without forcing a server restart.
<details>
<summary>Decision log</summary>
- The hash is intentionally not persisted in the database. The RFC discussed this trade-off; computing on demand keeps the schema minimal and lets the canonical shape evolve without a migration.
- The lock uses an `iota` slot in `coderd/database/lock.go` rather than `GenLockID` so it's stable, easy to audit, and matches the convention used for every other startup lock.
- A bearer-token Anthropic provider whose env vars also set Bedrock metadata but no AWS credentials does NOT store the Bedrock fields. Without credentials the discriminated settings would misrepresent the row as Bedrock auth.
- We deliberately do NOT publish to the `ai_providers_changed` pubsub channel from the seed because the seed completes before any subscriber is started; the follow-up PR introduces that channel.
</details>
Adds options matching new AI Gateway naming.
New options are added as alias for old options. Old options are still
working.
Old options have deprecated message.
No conflict detection was added.
Updated documentation so it mentions only new options. Added note about
old options still working.
> Various AI tools where used to create this PR
<!--
If you have used AI to produce some or all of this PR, please ensure you have read our [AI Contribution guidelines](https://coder.com/docs/about/contributing/AI_CONTRIBUTING) before submitting.
-->
relates to GRU-18
Adds basic implementation for Workspace Agent Connection Watch and tests.
Missing are handling of logs.
> Mux updated this PR on behalf of Mike.
## Stack Context
This PR is the API test coverage slice in the experimental personal
skills stack. The storage, schema, permissions, API, and SDK
implementation merged in #25363.
Stack order:
1. #25362 personal skill resolver
2. #25363 storage, permissions, API, and SDK
3. #25365 API test coverage
4. #25366 chattool and chatd integration
5. #25066 settings UI and docs
6. #25386 personal skills slash menu
## What?
Adds API and audit tests for personal skill CRUD, validation failures,
limits, authorization, soft-delete cleanup, and audit content tracking.
This PR is now test-only. It does not include migrations, generated
database code, or API implementation changes.
## Why?
The feature touches storage, permissions, and audit behavior. These
tests make the server behavior reviewable and protected without
re-reviewing the implementation that already merged in #25363.
## Validation
- `go test ./coderd -run '^(TestUserSkill|TestPatchUserSkill)' -count=1`
- `go test ./enterprise/coderd -run
'^TestUserSkillAuditDiffTracksContent$' -count=1`
- pre-commit hook via `gt modify --no-edit`
> Mux updated this PR on behalf of Mike.
## Stack Context
This PR is the storage, permissions, API, and SDK layer for experimental
personal skills. #25362 has landed on `main`, so this branch is
restacked directly on `main`.
Stack order:
1. #25363 storage, permissions, API, and SDK
2. #25365 API test coverage
3. #25366 chattool and chatd integration
4. #25066 settings UI and docs
5. #25386 personal skills slash menu
## What?
Adds the `user_skills` database table, generated queries, RBAC resources
and scopes, audit resource handling, experimental user-scoped CRUD
endpoints, SDK types, and generated API/site types.
Follow-up review and restack fixes:
- Enforce a bounded personal skill description in parser and database
constraints.
- Return `403 Forbidden` for unauthorized create and update attempts.
- Return explicit conflict responses when soft-deleted users are
targeted.
- Keep user admins out of personal skills, while site owners can read
and delete but not create or update.
- Document trigger-raised constraint names and keep schema constants
covered by tests.
- Reuse `UserSkillMetadata` in the full `UserSkill` SDK response type.
- Generate user skill IDs in Go instead of relying on a database
default.
- Rebase on latest `main` and renumber the user skills migration to
`000502_user_skills`.
## Why?
Personal skills need durable user-owned storage with owner
authorization, limited site-owner moderation, and a hidden API surface
before chatd can consume them.
## Validation
- `make gen`
- `go test ./coderd/database -run '^TestUserSkillSchemaConstants$'
-count=1`
- `go test ./coderd/database/dbauthz -run
'^TestMethodTestSuite/TestUserSkills$' -count=1`
- `go test ./coderd -run '^TestPatchUserSkill$' -count=1`
- `go test ./codersdk ./coderd/database/db2sdk`
- `make lint`
- pre-commit hook on `97fd58108d`
*Disclaimer: implemented by a Coder Agent using Claude Opus 4.6*
The `reconciliation stats` log line runs on every reconciliation tick
(every 5 minutes by default), even when there are no presets to
reconcile. In a steady-state installation without prebuilds activity,
this is the only log line that persistently shows up at info level.
Demote it to `debug` so the steady-state log output stays quiet.
Before:
```
2026-05-14 15:01:25.085 [info] coderd.prebuilds: reconciliation stats elapsed=1.649153ms presets_total=0 presets_reconciled=0
```
After: same line is emitted at `debug` level and is suppressed at the
default info log level.
This PR builds on top of https://github.com/coder/coder/pull/25070 to
add a way of running the larger "fake agent" manager via the existing
CLI, pulling in the URL/credentials already set.
With this, we can run a pod per scaletest region to act as all the
workspaces in that region.
This is in a new subcommand `scaletest agentfake` currently.
---------
Signed-off-by: Callum Styan <callumstyan@gmail.com>
These chatd tests are flaking for the same stale control-notification
race tracked by CODAGT-353, so this change skips the newly reflaking
advisor-chain and `TestPatchChatMessage/ChangesModel` tests and rewrites
the older `TODO(hugodutka)` skips to point at the same root cause. This
keeps the known flakes documented consistently until the chatd
notification-flow refactor lands.
Closes CODAGT-427
Closes https://github.com/coder/internal/issues/1510
This PR introduces a "fake agent" + manager, which can be used during
scaletests to run a single executable that acts as many workspace
agents. The goals of these are to provide a much lighter weight
implementation of a workspace in terms of resource cost and startup time when executing scaletests.
---------
Signed-off-by: Callum Styan <callumstyan@gmail.com>
Co-authored-by: Mux <noreply@coder.com>
Closes
https://linear.app/codercom/issue/AIGOV-284/add-group-budgets-table-and-crud-api
## Summary
Adds the `group_ai_budgets` table and the following endpoints:
- `GET /api/v2/groups/{group}/ai/budget`
- `PUT /api/v2/groups/{group}/ai/budget`
- `DELETE /api/v2/groups/{group}/ai/budget`
Each group may have at most one budget row. If no row exists, no budget
is enforced.
### Feature gate
Added `RequireFeatureMW(FeatureAIBridge)` on the `/ai/budget` sub-route.
## RBAC
Authorization reuses `rbac.ResourceGroup` with the existing
`.InOrganization(...).WithID(...)` scoping model.
The `dbauthz` wrappers load the parent `groups` row and authorize
against it.
No new resource type is introduced. As a result, anyone with
`group:update` permissions (Owner, OrgAdmin, or UserAdmin within the
organization) can manage AI budgets for that group.
## Read access for group members
`database.Group.RBACObject()` grants `policy.ActionRead` to all members
of the group through the group ACL:
```go
func (g Group) RBACObject() rbac.Object {
return rbac.ResourceGroup.WithID(g.ID).
InOrg(g.OrganizationID).
// Group members can read the group.
WithGroupACL(map[string][]policy.Action{
g.ID.String(): {
policy.ActionRead,
},
})
}
```
Because the `GET` endpoint authorizes against the same loaded `Group`
object, any group member can call:
```text
GET /api/v2/groups/{group}/ai/budget
```
`PUT` and `DELETE` remain admin-only. The group ACL grants only
`ActionRead`, so write operations continue to require role-based
`group:update` permissions.
## Alternative considered
A dedicated `rbac.ResourceGroupAiBudget` resource would allow budget
management to be separated from general group administration.
We decided not to add that complexity for now.
Closes https://github.com/coder/coder/issues/13112
**Breaking Change**: Removed status code `StatusNotModified` when no
diffs occur in a patch. Now the patch is always applied and a template
is always returned.
Extend the delete_deleted_user_resources() trigger so that secrets
belonging to a soft-deleted user are removed in the same transaction as
the existing api_keys and user_links cleanup.
user_secrets.user_id has ON DELETE CASCADE, but Coder soft-deletes users
by flipping users.deleted rather than removing the row, so the foreign key
cascade never fires and secrets would otherwise survive deletion.
Assisted by Coder Agents.
Use typed atomics (atomic.Int64, atomic.Int32, etc.) in test files to prevent
mixing atomic and non-atomic access on the same value, guarantee 64-bit
alignment on 32-bit platforms, and provide a cleaner API.
coder/fantasy now fails closed when Anthropic or OpenAI Responses
streams close before their provider terminal events instead of yielding
a successful finish.
This bumps the fantasy replacement to coder/fantasy#33 and teaches chat
error classification to treat those failures as retryable timeout errors
with explicit stream-closed messages.
<img width="875" height="311" alt="image"
src="https://github.com/user-attachments/assets/69c6f7b5-c885-46d2-a88b-b7a2b111bd55"
/>
## Description
Adds automatic key failover for centralized OpenAI provider, covering both chat completions and responses APIs. Same shape as the Anthropic PR: each upstream call walks the configured key pool, keys are marked **temporary** on 429 (with cooldown from `Retry-After`) and **permanent** on 401/403. Each agentic-loop iteration gets its own fresh walker so a tool-call continuation can fail over independently of the initial request.
BYOK is unchanged: BYOK requests run as a single attempt with no failover.
## Changes
- `config.OpenAI` carries a `KeyPool`. `Key` remains for BYOK Authorization Bearer set per interception.
- Chat completions blocking interceptor: walks the pool via `newChatCompletionWithKeyFailover`, marks keys on key-specific failures, returns on first success or non-failover error.
- Chat completions streaming interceptor: per-iteration walker. Pre-stream failures fail over to the next key; mid-stream errors are relayed as SSE events.
- Responses blocking interceptor: extracts `newResponseWithKeyFailover` parallel to chatcompletions.
- Responses streaming interceptor: per-iteration walker, retains the existing buffer-then-forward design.
## Related Issues
Related to: https://github.com/coder/internal/issues/1446
Related to: https://linear.app/codercom/issue/AIGOV-197/aibridge-automatic-key-failover-for-bridged-and-passthrough-routes
## Follow-up PRs
- Bedrock multi-key support.
- Refactor provider vs interceptor config separation.
- Record the actually-used key in the interception credential hint after failover.
> [!NOTE]
> Initially generated by Claude Opus 4.7, modified and reviewed by @ssncferreira
## Description
Adds automatic key failover for centralized Anthropic provider. When a key pool is configured, each upstream call walks the pool and tries keys in order until one succeeds or the pool is exhausted. Keys are marked **temporary** on 429 (with cooldown from `Retry-After`) and **permanent** on 401/403. Errors that aren't key-specific don't trigger failover. Each agentic-loop iteration gets its own fresh walker, so a tool-call continuation can fail over independently of the initial request.
BYOK is unchanged: BYOK requests run as a single attempt with no failover.
## Changes
- `config.Anthropic` carries a `KeyPool`. `Key` remains for BYOK X-Api-Key set per interception.
- Blocking interceptor: walks the pool, marks keys on key-specific failures, returns on first success or non-failover error.
- Streaming interceptor: per-iteration walker. Pre-stream failures fail over to the next key; mid-stream errors are relayed as SSE events.
- New `keypool` error types: `TransientExhaustionError` (carries soonest cooldown) and `ErrPermanentExhaustion`. Replace the prior `ErrAllKeysExhausted`.
- Error responses now consistently include the outer `"type": "error"` field.
## Related Issues
Related to: https://github.com/coder/internal/issues/1446
Related to: https://linear.app/codercom/issue/AIGOV-197/aibridge-automatic-key-failover-for-bridged-and-passthrough-routes
## Follow-up PRs
- Bedrock multi-key support.
- Refactor provider vs interceptor config separation.
- Record the actually-used key in the interception credential hint after failover.
> [!NOTE]
> Initially generated by Claude Opus 4.7, modified and reviewed by @ssncferreira
Persists the agent-generated turn-end summary on `chats` and shows it as
the Agents sidebar subtitle when present, falling back to the model
name. Errors still take precedence.
> Mux is acting on Mike's behalf.
## What changes
**Storage.** New nullable `last_turn_summary` column on `chats`
(migration `000486`). New `UpdateChatLastTurnSummary` query normalizes
blank/whitespace input to `NULL`, preserves `updated_at` (so the chat
does not jump to the top of the sidebar on summary writes), and uses an
`expected_updated_at` stale-write guard so an older async summary cannot
overwrite a newer turn.
**Backend.** `coderd/x/chatd/chatd.go` decouples summary generation from
webpush. Generated summaries persist for completed parent turns even
when webpush is unconfigured or has no subscriptions. The same generated
text is reused as the webpush body when webpush is configured, so the
summary model is not called twice. Generic fallback push text is no
longer persisted; it clears any stale summary instead.
Error/interrupt/pending-action terminal paths clear `last_turn_summary`
for the latest turn.
**Frontend.** `AgentsSidebar.tsx` subtitle priority is now `errorReason
|| lastTurnSummary || modelName`, normalized via the existing
`asNonEmptyString` helper from `blockUtils.ts`.
## Tests
- `TestUpdateChatLastTurnSummary` (database): success,
whitespace-to-NULL, stale guard rejects, `updated_at` preserved.
- `TestUpdateLastTurnSummaryRejectsStaleWrites` (chatd internal): direct
stale-`expected_updated_at` test.
- `TestSuccessfulChatPersistsTurnSummaryWithoutWebPush`: persistence
works without webpush subscriptions.
- `TestSuccessfulChatSendsWebPushWithSummary`: same generated text
drives both DB and push body.
-
`TestSuccessfulChatSendsWebPushFallbackWithoutSummaryForEmptyAssistantText`:
fallback text is not persisted.
- `TestErroredChatClearsLastTurnSummaryAndSendsWebPush`: error path
clears the field.
- `TestInterruptChatDoesNotSendWebPushNotification`: interrupt path
clears the field, no push fires.
- `AgentsSidebar.test.tsx`: subtitle priority for summary-present,
error-wins, no-summary fallback, whitespace fallback.
- `AgentsSidebar.stories.tsx`: `ChatWithTurnSummary` and
`ChatWithTurnSummaryAndError`.
## Notes
- No backfill. Existing chats keep showing the model name until their
next turn completes.
- Parent chats only in this iteration; the field is rendered on any
`Chat` if a future change extends generation to children.
- Decoupling generation from webpush adds quickgen model calls for
completed parent turns that previously skipped generation when no
subscriptions existed. Existing parent-only, assistant-text-present,
`PushSummaryModel` configured, and bounded-timeout gates keep this
behavior bounded.
Tightens the chat stream subscription path on a few related axes. None
of these changes touch the steady-state event flow; they all concern the
subscribe handshake.
## Motivation
`Server.Subscribe` carries three responsibilities that were entangled:
1. Authorize the caller against the chat row.
2. Arm local + pubsub subscriptions before any DB reads
(subscribe-first-then-query).
3. Build the initial snapshot from a fresh chat row, message history,
and queue.
When all three live in one function and share the request context, a few
unfortunate behaviors fall out:
- The HTTP handler's middleware already loaded and authorized the chat
row, but `Subscribe(chatID)` discarded it and re-fetched on every
WebSocket connection.
- The chat row used to populate the initial `status` event was loaded
*before* the pubsub subscription was armed, so a status transition that
happened in that window was silently lost.
- Control-path DB reads inherited whatever context the caller passed in.
A caller without a deadline could wedge a subscriber goroutine
indefinitely on a stalled DB.
- A transient failure of the chat re-read collapsed the entire
subscription instead of degrading gracefully.
## What changes
**Split the auth boundary out into the type signature.** A new
`SubscribeAuthorized(ctx, chat, ...)` takes the already-authorized row
directly. The HTTP handler in `coderd/exp_chats.go` calls it with the
chat row from `httpmw.ChatParam`, eliminating the redundant
`GetChatByID`. `Subscribe(chatID)` is preserved as a thin wrapper for
callers that don't have a chat row in hand (tests, internal callers); it
does the auth lookup and delegates.
**Re-read the chat after arming subscriptions.** Inside
`SubscribeAuthorized`, after the local stream and pubsub subscriptions
are active, we reload the chat row to populate the initial `status`
event and any enterprise relay setup. Combined with the existing
subscribe-first-then-query pattern, this closes the gap where a status
transition between the middleware's load and the subscription arming
would not appear in either the initial snapshot or a live notification.
**Fall back to the middleware row on refresh failure.** If the
post-subscription refresh fails (transient DB blip, brief pool
exhaustion), we log a warning and reuse the row that proved
authorization in the first place. Messages, queue, and pubsub are all
independent of this row, so the stream still works; the initial `status`
is just slightly stale and self-corrects via the next pubsub event.
**Bound subscriber control-path DB reads.** A new
`streamSubscriberControlFetchContext` helper applies a 5-second fallback
timeout only when the caller has no deadline of their own. Used at the
chat refresh, the initial queue load, and the queue-update goroutine
following pubsub notifications. HTTP-driven callers pass through
unchanged; background callers can no longer hang forever on a stalled DB
and leak subscriber goroutines, pubsub subscriptions, and `chatStreams`
entries.
Moves the chat error kind taxonomy from `coderd/x/chatd/chaterror` into
`codersdk.ChatErrorKind` and types `ChatError.Kind` /
`ChatStreamRetry.Kind` so generated TypeScript exposes an SDK-owned
union, including `usage_limit`. Backend chat classification now
references the SDK constants directly while preserving the existing JSON
string values.
Keeps chat usage-limit admission failures on their existing 409 response
shape. The frontend maps structured usage-limit responses to the
SDK-owned `usage_limit` kind, uses generated `TypesGen.ChatErrorKind`
directly, and removes the local string union and alias.
Drop start, lastWrite, and overwrites fields on connIO along with the
Stats() and Overwrites() methods. They have had no readers since
52901e121 which rewrote the PG coordinator's debug page to query the
database directly.
**Breaking change for changelog:**
> `codersdk.Chat.last_error` now returns a structured `ChatError` object
(`{message, kind, provider, retryable, status_code, detail}`) instead of
a plain string. The chats API is experimental
(`/api/experimental/chats`), so this ships without a deprecation cycle;
consumers reading `chat.last_error` as a string must update to read
`chat.last_error.message`. SDK/generated TypeScript terminal error
payloads now use the single `ChatError` type; the live stream error
payload type is renamed from `ChatStreamError` to `ChatError`.
Persisted chat errors now carry the same provider-specific detail (kind,
provider, retryable, HTTP status, optional detail) as the live stream,
so refreshing a failed chat rehydrates with the full structured error
instead of a one-line headline.
Existing rows are migrated in place: legacy text errors are wrapped into
`{message, kind: "generic"}` so already-errored chats still render, and
rows with `last_error IS NULL` stay NULL. Internally, persisted fallback
decoding now reuses the existing `chaterror.KindGeneric` constant, with
no JSON value change.
Closes CODAGT-239
Danny Kopping
