Add Unwrap() to StatusWriter so http.ResponseController.SetWriteDeadline
can reach the underlying net.Conn through the middleware wrapper. Without
this, the agent's 20s WriteTimeout killed blocking process output
connections.
Also add 30s headroom to the write deadline in handleProcessOutput so
the response can be written after a full-duration blocking wait.
On the tool layer, waitForProcess and the process_output tool now try a
non-blocking snapshot on any error, not just context timeout. Transport
errors (like the WriteTimeout EOF) previously returned with no process
ID and no recovery path. Now if the process finished, the result is
returned transparently. If still running, the error includes the process
ID and tells the agent to use process_output.
- Add `agents_workspace_ttl` site config (default: whatever the template
says a.k.a. `0s`)
- Expose via GET/PUT `/api/experimental/chats/config/workspace-ttl`
- Chat tool reads setting and passes `TTLMillis` on workspace creation
- Existing autostop infrastructure handles the rest (zero changes to
LifecycleExecutor, CalculateAutostop, or activity bumping)
- ⚠️ Template-level `UserAutostopEnabled=false` overrides this global
default. Not touching this.
- Frontend: "Workspace Lifetime" control in /agents/settings Behavior
tab (admin-only)
> This PR was created with the help of Coder Agents, and has been
reviewed by several humans and robots. 🤖🤝🧑💻
## Summary
Adds a new `coderd/chatd/mcpclient` package that connects to
admin-configured MCP servers and wraps their tools as
`fantasy.AgentTool` values that the chat loop can invoke.
## What changed
### New: `coderd/chatd/mcpclient/mcpclient.go`
The core package with a single entry point:
```go
func ConnectAll(
ctx context.Context,
logger slog.Logger,
configs []database.MCPServerConfig,
tokens []database.MCPServerUserToken,
) (tools []fantasy.AgentTool, cleanup func(), err error)
```
This:
1. Connects to each enabled MCP server using `mark3labs/mcp-go`
(streamable HTTP or SSE transport)
2. Discovers tools via the MCP `tools/list` method
3. Wraps each tool as a `fantasy.AgentTool` with namespaced name
(`serverslug__toolname`)
4. Applies tool allow/deny list filtering from the server config
5. Handles auth: OAuth2 bearer tokens, API keys, and custom headers
6. Skips broken servers with a warning (10s connect timeout per server)
7. Returns a cleanup function to close all MCP connections
### Modified: `coderd/chatd/chatd.go`
In `runChat()`, after loading the model/messages but before assembling
the tool list:
- Reads `chat.MCPServerIDs` from the chat record
- Loads the MCP server configs from the database
- Resolves the user's auth tokens
- Calls `mcpclient.ConnectAll()` to connect and discover tools
- Appends the MCP tools to the chat's tool set
- Defers cleanup to close connections when the chat turn ends
The chat loop (`chatloop.Run`) already handles tools generically —
MCP-backed tools are invoked identically to built-in workspace tools. No
changes needed in `chatloop/`.
### New: `coderd/chatd/mcpclient/mcpclient_test.go`
10 tests covering:
- Tool discovery and namespacing
- Tool call forwarding and result conversion
- Allow/deny list filtering
- Connection failure handling (graceful skip)
- Multi-server support with correct prefixes
- OAuth2 auth header injection
- Disabled server skipping
- Invalid input handling
- Tool info parameter propagation
## Design decisions
- **Tool namespacing**: `slug__toolname` with double underscore
separator. Avoids collisions with tools containing single underscores.
Stripped when forwarding to `tools/call`.
- **Connection lifecycle**: Fresh connections per chat turn, closed via
`defer`. Matches the `turnWorkspaceContext` pattern.
- **Failure isolation**: Each server connects independently. A broken
server doesn't fail the chat — its tools are simply unavailable.
- **No chatloop changes**: The existing `[]fantasy.AgentTool` interface
is already fully generic.
## What's NOT in this PR (follow-ups)
- Frontend MCP server picker UI (selecting servers for a chat)
- System prompt additions describing available MCP tools
- Token refresh on expiry mid-chat
- The deprecated `aibridged` MCP proxy cleanup
> **PR Stack**
> 1. #23351 ← `#23282`
> 2. #23282 ← `#23275`
> 3. #23275 ← `#23349`
> 4. **#23349** ← `main` *(you are here)*
---
Retry events were published only to the local in-process stream via
`publishEvent()`. When pubsub is active, `Subscribe()`'s merge loop only
forwarded durable events (messages, status, errors) from pubsub
notifications,
so retry events were silently dropped for cross-replica subscribers.
This adds a `publishRetry()` helper that publishes both locally and via
pubsub,
and extends the `Subscribe()` notification handler to forward retry
events.
**Changes:**
- `coderd/pubsub/chatstreamnotify.go`: add `Retry` field to notify
message
- `coderd/chatd/chatd.go`: add `publishRetry()`, update `OnRetry`
callback,
extend `Subscribe()` to forward `notify.Retry`
- `coderd/chatd/chatd_internal_test.go`: focused pubsub delivery test
- `enterprise/coderd/chatd/chatd_test.go`: cross-replica end-to-end test
Audited exported helpers in `coderd/util/*`, `testutil`, `cryptorand`,
and friends, then replaced duplicated implementations with canonical
versions.
- **fix: `maps.SortedKeys` generic signature** — value type was
hardcoded to `any`, making it impossible to actually call. Added second
type parameter `V any`. Added table-driven tests with `cmp.Diff`.
- **refactor: replace ad-hoc ptr helpers with `ptr.Ref`** — removed
`int64Ptr`, `stringPtr`, `boolPtr`, `i64ptr`, `strPtr`, `PtrInt32`
across 6 files.
- **refactor: replace local `sortedKeys`/`sortKeys` with
`maps.SortedKeys`** — now that the signature is fixed, scripts can use
it.
- **refactor: replace hand-rolled `capitalize` with
`strings.Capitalize`** — the typegen version was also not UTF-8 safe.
> 🤖 This PR was created with the help of Coder Agents, and was reviewed
by my human. 🧑💻
Replace the 200ms polling loop in chatd's execute and
process_output tools with server-side blocking via sync.Cond
on HeadTailBuffer.
The agent's GET /{id}/output endpoint accepts ?wait=true to
block until the process exits or a 5-minute server cap expires.
The process_output tool blocks by default for 10s (overridable
via wait_timeout), and falls back to a non-blocking snapshot on
timeout. The execute tool's foreground path makes a single
blocking call instead of polling.
Related #23316
- Rate-limit "chat stream buffer full" and "dropping chat stream event"
WARN logs to at most once per 10s per chat.
- Intermediate drops not logged; WARN includes `dropped_count`.
- Per-chat tracking on `chatStreamState` using timestamp comparison
against `quartz.Clock` — no global tickers, no new `Server` fields.
- Subscriber and buffer drop counters reset at all lifecycle boundaries.
> 🤖 This PR was created with the help of Coder Agents, and was reviewed
by my human. 🧑💻
The ampersand detection treated bash's pipe-stderr operator (|&)
as a trailing & for backgrounding, stripping it and producing a
broken pipe command. Also adds tests for execute.go and chatloop
context limit helpers, covering previously untested edge cases.
- Wire `workspacestats.ActivityBumpWorkspace` into `trackWorkspaceUsage`
so the workspace build deadline is extended each time the chat heartbeat
fires
- Prevents mid-conversation autostop for chat workspaces
- Updates `TestHeartbeatBumpsWorkspaceUsage` verifying the deadline bump
> This PR was created with the help of Coder Agents, and was reviewed by two humans and their pet robots 🧑💻🤝🤖
The processChat defer at line 2464 catches panics on its main
goroutine and transitions the chat to error status. This was
previously untested.
The test wraps the database Store to panic during PersistStep's
InTx call, which runs synchronously on the processChat goroutine.
A tool-level panic wouldn't work because executeTools has its own
recover that converts panics into tool error results.
## Problem
Anthropic rejects requests containing empty text content blocks with:
```
messages: text content blocks must be non-empty
```
Empty text parts (`""` or whitespace-only like `" "`) get persisted in
the database when a stream sends `TextStart`/`TextEnd` with no
`TextDelta` in between. On the next turn, these parts are loaded from
the DB and sent to Anthropic, which rejects them.
## Fix
Filter empty/whitespace-only text and reasoning parts at the two LLM
dispatch boundaries, without modifying persistence (the raw record is
preserved):
- **`partsToMessageParts()`** in `chatprompt.go` — filters when
converting persisted DB messages to fantasy message parts for LLM calls.
This is the last gateway before the Anthropic provider creates
`TextBlockParam` objects.
- **`toResponseMessages()`** in `chatloop.go` — filters when building
in-flight conversation messages between steps within a single turn.
Note: `flushActiveState()` (the interruption path) already had this
guard — the normal `TextEnd` streaming path did not, but since we're not
changing persistence, the fix is applied at the dispatch layer.
## Problem
When `Store: true` is set for OpenAI Responses API calls (the new
default), multi-turn conversations with reasoning models fail on the
second message:
```
stream response: bad request: Item 'rs_xxx' of type 'reasoning' was provided
without its required following item.
```
The fantasy library was reconstructing full `OfReasoning` input items
(with encrypted content and summary) when replaying assistant messages.
The API cannot pair these reconstructed reasoning items with the output
items that originally followed them because the output items are sent as
plain `OfMessage` without server-side IDs.
## Fix
Updates the fantasy dependency (`kylecarbs/fantasy@cj/go1.25`) to skip
reasoning parts during conversation replay in `toResponsesPrompt`. With
`Store` enabled, the API already has the reasoning persisted server-side
— it doesn't need to be replayed in the input.
Fantasy PR: https://github.com/charmbracelet/fantasy/pull/181
## Testing
Adds `TestOpenAIReasoningRoundTrip` integration test that:
1. Sends a query to `o4-mini` (reasoning model with `Store: true`)
2. Verifies reasoning content is persisted
3. Sends a follow-up message — this was the failing step
4. Verifies the follow-up completes successfully
Requires `OPENAI_API_KEY` env var to run.
Eliminates the timing flake in
`TestInterruptAutoPromotionIgnoresLaterUsageLimitIncrease` by making the
chatd worker loop clock-controllable.
## Changes
**`coderd/chatd/chatd.go`**
- Replace `time.NewTicker` calls in `Server.start()` with
`p.clock.NewTicker` using named quartz tags `("chatd", "acquire")` and
`("chatd", "stale-recovery")`.
**`coderd/chatd/chatd_test.go`**
- Inject `quartz.NewMock(t)` into the test via `newActiveTestServer`
config override.
- Trap the acquire ticker so the test controls exactly when pending
chats are reacquired.
- Rewrite the test flow as explicit clock-advance steps instead of
wall-clock polling.
**`AGENTS.md`**
- Document the PR title scope rule (scope must be a real path containing
all changed files).
## Validation
- `go test ./coderd/chatd -run
TestInterruptAutoPromotionIgnoresLaterUsageLimitIncrease -count=100` ✅
- `go test ./coderd/chatd` ✅
- `make lint` ✅
## Summary
Adds the database schema, API endpoints, SDK types, and encryption
wrappers for admin-managed MCP (Model Context Protocol) server
configurations that chatd can consume. This is the backend foundation
for allowing external MCP tools (Sentry, Linear, GitHub, etc.) to be
used during AI chat sessions.
## Database
Two new tables:
- **`mcp_server_configs`**: Admin-managed server definitions with URL,
transport (Streamable HTTP / SSE), auth config (none / OAuth2 / API key
/ custom headers), tool allow/deny lists, and an availability policy
(`force_on` / `default_on` / `default_off`). Includes CHECK constraints
on transport, auth_type, and availability values.
- **`mcp_server_user_tokens`**: Per-user OAuth2 tokens for servers
requiring individual authentication. Cascades on user/config deletion.
New column on `chats` table:
- **`mcp_server_ids UUID[]`**: Per-chat MCP server selection, following
the same pattern as `model_config_id` — passed at chat creation,
changeable per-message with nil-means-no-change semantics.
## API Endpoints
All routes are under `/api/experimental/mcp/servers/` and gated behind
the `agents` experiment.
**Admin endpoints** (`ResourceDeploymentConfig` auth):
- `POST /` — Create MCP server config
- `PATCH /{id}` — Update MCP server config (full-replace)
- `DELETE /{id}` — Delete MCP server config
**Authenticated endpoints** (all users, enabled servers only for
non-admins):
- `GET /` — List configs (admins see all, members see enabled-only with
admin fields redacted)
- `GET /{id}` — Get config by ID (with `auth_connected` populated
per-user)
**OAuth2 per-user auth flow:**
- `GET /{id}/oauth2/connect` — Initiate OAuth2 flow (state cookie CSRF
protection)
- `GET /{id}/oauth2/callback` — Handle OAuth2 callback, store tokens
- `DELETE /{id}/oauth2/disconnect` — Remove stored OAuth2 tokens
## Security
- **Secrets never returned**: `OAuth2ClientSecret`, `APIKeyValue`, and
`CustomHeaders` are never in API responses — only boolean indicators
(`has_oauth2_secret`, `has_api_key`, `has_custom_headers`).
- **Field redaction for non-admins**: `convertMCPServerConfigRedacted`
strips `OAuth2ClientID`, auth URLs, scopes, and `APIKeyHeader` from
non-admin responses.
- **dbcrypt encryption at rest**: All 5 secret fields use `dbcrypt_keys`
encryption with full encrypt-on-write / decrypt-on-read wrappers (11
dbcrypt method overrides + 2 helpers), following the same pattern as
`chat_providers.api_key`.
- **OAuth2 CSRF protection**: State parameter stored in `HttpOnly`
cookie with `HTTPCookies.Apply()` for correct `Secure`/`SameSite` behind
TLS-terminating proxies.
- **dbauthz authorization**: All 18 querier methods have authorization
wrappers. Read operations use `ActionRead`, write operations use
`ActionUpdate` on `ResourceDeploymentConfig`.
## Governance Model
| Control | Implementation |
|---------|---------------|
| **Global kill switch** | `enabled` defaults to `false` |
| **Availability policy** | `force_on` (always injected), `default_on`
(pre-selected), `default_off` (opt-in) |
| **Per-chat selection** | `mcp_server_ids` on `CreateChatRequest` /
`CreateChatMessageRequest` |
| **Auth gate** | OAuth2 servers require per-user auth before tools are
injected |
| **Tool-level allow/deny** | Arrays on `mcp_server_configs` for
granular tool filtering |
| **Secrets encrypted at rest** | Uses `dbcrypt_keys` (same pattern as
`chat_providers.api_key`) |
## Tests
8 test functions covering:
- Full CRUD lifecycle (create, list, update, delete)
- Non-admin visibility filtering (enabled-only, field redaction)
- `auth_connected` population for OAuth2 vs non-OAuth2 servers
- Availability policy validation (valid values + invalid rejection)
- Unique slug enforcement (409 Conflict)
- OAuth2 disconnect idempotency
- Chat creation with `mcp_server_ids` persistence
## Known Limitations (Deferred)
These are documented and intentional for an experimental feature:
- **Audit logging** not yet wired — will add when feature stabilizes
- **Cross-field validation** (e.g., OAuth2 fields required when
`auth_type=oauth2`) — admin-only endpoint, will add when stabilizing
- **`force_on` auto-injection** — query exists but not yet wired into
chatd tool injection (follow-up)
- **Additional test coverage** — 403 auth tests, GET-by-ID tests,
callback CSRF tests planned for follow-up
## What's NOT in this PR
- Frontend UI (admin panel + chat picker)
- Actual MCP client connections (`chatd/chatmcp/` manager)
- Tool injection into `chatloop/`
Updates the `charm.land/fantasy` replace to the rebased `cj/go1.25`
branch on `kylecarbs/fantasy`, which now includes:
- **chore: downgrade to Go 1.25**
- **feat: anthropic computer use**
- **chore: use kylecarbs/openai-go fork for coder/coder compat**
Switches the `openai-go/v3` replace from `SasSwart/openai-go` →
`kylecarbs/openai-go`, which is the same SasSwart perf fork plus a fix
for `WithJSONSet` being clobbered by deferred body serialization.
Without the fix, `NewStreaming` silently drops `stream: true` from
requests. See https://github.com/kylecarbs/openai-go/pull/2 for details.
## Problem
Scaletest follow-up storms showed that the chat stream path was doing a
same-replica DB reread for every durable message it had already
delivered locally.
In a 600-chat / 10-turn run, `/stream`-attributed
`GetChatMessagesByChatID` calls reached about 14.2k across 5,400
follow-up turns — roughly **2.63 rereads per turn**. The primary coderd
replicas saturated their DB pools at 60/60 open connections during the
storm window.
The root cause: when pubsub was active, `Subscribe()` suppressed local
durable `message` events and relied entirely on pubsub notify →
`GetChatMessagesByChatID` for catch-up. Same-replica subscribers paid
the full DB round-trip even though the persisting process was on the
same replica.
## Solution
Add a bounded per-chat **durable message cache** to `chatStreamState` so
that same-replica subscribers can catch up from memory instead of the
database.
### How it works
1. `publishMessage()` caches the SDK event in `chatStreamState` before
local fanout and pubsub notify.
2. `publishEditedMessage()` replaces the cache with only the edited
message, then publishes `FullRefresh`.
3. `Subscribe()` handles ordinary `AfterMessageID` notifies by first
consulting the per-chat durable cache and only falling back to
`GetChatMessagesByChatID` on cache miss.
4. `FullRefresh` always forces a DB reread (cache is bypassed).
### Safety properties
- If the cache misses (e.g. message expired or remote replica), the DB
catch-up still runs — no silent message loss.
- `FullRefresh` (edits) always rereads from the database.
- Remote replicas still use the pubsub + DB path unchanged.
- The cache is bounded (`maxDurableMessageCacheSize = 256`) and scoped
per chat — no unbounded memory growth.
## Impact
This change removes the entire same-replica portion of the stream
rereads. Based on the 600-chat follow-up run, the upper bound on saved
work is the same-replica share of about 14.2k `GetChatMessagesByChatID`
rereads, with the observed total stream reread rate at about 2.63
rereads per follow-up turn.
This PR fixes a bug where if a tool result contained binary data it
wouldn't be persisted to the database.
`jsonb` in Postgres is unable to store null bytes which are sometimes
output by tool results. This change makes it so that we encode them with
a special escape sequence before saving them to the database, and decode
them on read.
<img width="808" height="637" alt="Screenshot 2026-03-11 at 13 14 06"
src="https://github.com/user-attachments/assets/9be353eb-ff26-40ec-9f0a-195022b11f43"
/>
- coderd: Wires `options.WorkspaceUsageTracker` into the chatd config.
- chatd: Adds `UsageTracker` and calls `UsageTracker.Add(workspaceID)`
on each heartbeat tick
- chatd: adds tests to verify `last_used_at` bump behaviour
> 🤖 This PR was created with the help of Coder Agents, and will be
reviewed by my human. 🧑💻
Adds a `deleted` boolean column to the `chat_messages` table. Messages
are never physically deleted from the database — instead they are marked
as deleted so that usage and cost data is preserved.
## Changes
### Migration
- New migration (000444) adds `deleted boolean NOT NULL DEFAULT false`
to `chat_messages`
### SQL queries
- `DeleteChatMessagesAfterID` → `SoftDeleteChatMessagesAfterID` (UPDATE
SET deleted=true instead of DELETE)
- New `SoftDeleteChatMessageByID` query for single-message soft-delete
- All read queries now filter `deleted = false`:
- `GetChatMessageByID`
- `GetChatMessagesByChatID`
- `GetChatMessagesByChatIDDescPaginated`
- `GetChatMessagesForPromptByChatID` (both CTE and main query)
- `GetLastChatMessageByRole`
- Cost/usage queries (`GetChatCostSummary`, `GetChatCostPerModel`, etc.)
intentionally still include deleted messages to preserve accurate spend
tracking
### EditMessage behavior
- Previously: updated the message content in-place + hard-deleted
subsequent messages
- Now: soft-deletes the original message + soft-deletes subsequent
messages + inserts a new message with the updated content
- This preserves the original message data (tokens, cost, content) in
the database
Follow-up to #23220, addressing Cian's review comments:
- **SQL casing**: Uppercase `UNNEST` to match `NULLIF`/`COALESCE`
convention in the query.
- **Builder pattern**: `chatMessage` struct now uses unexported fields
with a `newChatMessage` constructor for required fields (role, content,
visibility, modelConfigID, contentVersion) and chainable builder methods
(`withCreatedBy`, `withCompressed`, `withUsage`, `withContextLimit`,
`withTotalCostMicros`, `withRuntimeMs`) for optional/nullable fields.
- **Batch test in chats_test**: Replaced the `for i := 0; i < 2` loop
with a single batch insert of 2 messages to actually exercise the batch
logic.
- **Multi-message querier test**: Added `BatchInsertMultipleMessages`
test verifying 3-message batch insert with role ordering, sequential
IDs, nullable field semantics (NULL for zero UUIDs and zero ints), and
token/cost assertions.
---------
Co-authored-by: Cian Johnston <cian@coder.com>
Replaces the singular `InsertChatMessage` query with
`InsertChatMessages` that uses PostgreSQL's `unnest()` for batch
inserts. This reduces the number of database round-trips when inserting
multiple messages in a single transaction.
## Changes
- **SQL**: New `InsertChatMessages :many` query using `unnest()` arrays
following the existing codebase pattern (e.g.,
`InsertWorkspaceAgentStats`). Preserves the CTE that updates
`chats.last_model_config_id` using the last non-null model config from
the batch. Uses `NULLIF` for UUID columns to handle NULL foreign keys.
- **Go layers**: Updated `querier.go`, `dbauthz.go`,
`dbmetrics/querymetrics.go`, `dbmock/dbmock.go`, and `queries.sql.go` to
use the new batch signature (`[]ChatMessage` return type, array params).
- **chatd.go**: All call sites converted to batch inserts:
- **CreateChat**: System prompt + user message batched into one call
- **persistStep**: Assistant message + tool messages batched into one
call
- **persistSummary**: Hidden summary + assistant + tool messages batched
into one call
- Single-message sites use the same API with single-element arrays
- **Helper**: New `appendChatMessage` function simplifies building batch
params at each call site.
- **Tests**: All test files updated to use the new API.
Builds on top of #23213.
## Summary
Adds a `runtime_ms` column to `chat_messages` that records the
wall-clock duration (in milliseconds) of each LLM step. This covers LLM
streaming, tool execution, and retries — the full time the agent is
"alive" for a step.
This is the foundation for billing by agent alive time. The column
follows the same pattern as `total_cost_micros`: stored per assistant
message, aggregatable with `SUM()` over time periods by user.
## Changes
- **Migration**: adds nullable `runtime_ms bigint` to `chat_messages`.
- **chatloop**: adds `Runtime time.Duration` field to `PersistedStep`,
measures `time.Since(stepStart)` at the beginning of each step (covering
stream + tool execution + retries).
- **chatd**: passes `step.Runtime.Milliseconds()` to the assistant
message `InsertChatMessage` call; all other message types (system, user,
tool) get `NULL`.
- **Tests**: adds `runtime > 0` assertion in chatloop tests.
## Billing query pattern
Once ready, aggregation mirrors the existing cost queries:
```sql
SELECT COALESCE(SUM(cm.runtime_ms), 0)::bigint AS total_runtime_ms
FROM chat_messages cm
JOIN chats c ON c.id = cm.chat_id
WHERE c.owner_id = @user_id
AND cm.created_at >= @start_time
AND cm.created_at < @end_time
AND cm.runtime_ms IS NOT NULL;
```
When a chat is created via `chatd`, a system message is now inserted
informing the model whether the chat was created with or without a
workspace.
**With workspace:**
> This chat is attached to a workspace. You can use workspace tools like
execute, read_file, write_file, etc.
**Without workspace:**
> There is no workspace associated with this chat yet. Create one using
the create_workspace tool before using workspace tools like execute,
read_file, write_file, etc.
This is a model-only visibility system message (not shown to users) that
helps the model understand its available capabilities upfront —
particularly important for subagents spawned without a workspace, which
previously would attempt to use workspace tools and fail.
**Changes:**
- `coderd/chatd/chatd.go`: Added workspace awareness constants and
inserted the system message in `CreateChat` after the system prompt,
before the initial user message.
- `coderd/chatd/chatd_test.go`: Added
`TestCreateChatInsertsWorkspaceAwarenessMessage` with sub-tests for both
with-workspace and without-workspace cases.
## Summary
- add a hidden deployment config option for chat acquire batch size
(`CODER_CHAT_ACQUIRE_BATCH_SIZE` / `chat.acquireBatchSize`)
- thread the configured value into chatd startup while preserving the
existing default of `10`
- clamp the deployment value to the `int32` range before passing it into
chatd
- regenerate the API/docs/types/testdata artifacts for the new config
field
## Why
`chatd` currently acquires pending chats in batches of `10` via a
compile-time default. This change makes that batch size
operator-configurable from deployment config, so we can tune acquisition
behavior without another code change.
## Problem
When a user cancels a streaming chat response mid-stream, the partial
content disappears entirely — both from the UI and the database. The
streamed text vanishes as if the response never happened.
## Root Causes
Three issues combine to prevent partial message persistence on
interrupt:
### 1. StreamPartTypeError only matched `context.Canceled`
(`chatloop.go`)
The interrupt detection in `processStepStream` checked:
```go
errors.Is(part.Error, context.Canceled) && errors.Is(context.Cause(ctx), ErrInterrupted)
```
But some providers propagate `ErrInterrupted` directly as the stream
error rather than wrapping it in `context.Canceled`. This caused the
condition to fail, so `flushActiveState` was never called and partial
text accumulated in `activeTextContent` was lost.
### 2. No post-loop interrupt check (`chatloop.go`)
If the stream iterator stops yielding parts without producing a
`StreamPartTypeError` (e.g., a provider that silently closes the
response body on cancel), there was no check after the `for part :=
range stream` loop to detect the interrupt and flush active state.
### 3. Worker ownership check blocked interrupted persists (`chatd.go`)
`InterruptChat` → `setChatWaiting` clears `worker_id` in the DB
**before** the chatloop detects the interrupt. When
`persistInterruptedStep` (using `context.WithoutCancel`) tried to write
the partial message, the ownership check:
```go
if !lockedChat.WorkerID.Valid || lockedChat.WorkerID.UUID != p.workerID {
return chatloop.ErrInterrupted // always blocks!
}
```
unconditionally rejected the write. The error was silently logged as a
warning.
## Fix
- **Broaden the `StreamPartTypeError` interrupt detection** to match
both `context.Canceled` and `ErrInterrupted` as the stream error.
- **Add a post-loop interrupt check** in `processStepStream` that
flushes active state when the context was canceled with
`ErrInterrupted`.
- **Allow `persistStep` to write when the chat is in `waiting` status**
(interrupt) even if `worker_id` was cleared. The `pending` status (from
`EditMessage`, where history is truncated) still correctly blocks stale
writes.
## Testing
Added `TestInterruptChatPersistsPartialResponse` — an end-to-end
integration test that:
1. Streams partial text chunks from a mock LLM
2. Waits for the chatloop to publish `message_part` events (confirming
chunks were processed)
3. Interrupts the chat mid-stream
4. Verifies the partial assistant message is persisted in the database
with the expected text content
Adds a new `site_config` entry that controls whether the virtual desktop
feature for Coder Agents is enabled. It can be set via a new
`/api/experimental/chats/config/desktop-enabled` endpoint, which will be
used by the frontend.
## Summary
Two targeted performance improvements to the chatd server, identified
through benchmarking.
### 1. RWMutex for instruction cache
The instruction cache is read on every chat turn to fetch the home
instruction file for a workspace agent. Writes only occur on cache
misses (once per agent per 5-minute TTL window), making the access
pattern ~90%+ reads.
Switching from `sync.Mutex` to `sync.RWMutex` and using
`RLock`/`RUnlock` on the read path allows concurrent readers instead of
serializing them.
**Benchmark (200 concurrent chats):**
| | ns/op |
|---|---|
| Mutex | 108 |
| RWMutex | 32 |
| **Speedup** | **3.4x** |
### 2. Hoist JSON marshaling out of persistStep transaction
`MarshalParts`, `PartFromContent`, `CalculateTotalCostMicros`, and the
`usageForCost` struct population are pure CPU work that ran inside the
`FOR UPDATE` transaction in `persistStep`. They have zero dependency on
the database transaction.
Moving all marshal and cost-calculation calls above `p.db.InTx()` means
the row lock is held only for `GetChatByIDForUpdate` +
`InsertChatMessage` calls.
**Benchmark (16 goroutines contending on same lock):**
| Tool calls | Inside lock | Outside lock | Speedup |
|---|---|---|---|
| 1 | 13,977 ns/op | 1,055 ns/op | 13x |
| 5 | 38,203 ns/op | 3,769 ns/op | 10x |
| 10 | 67,353 ns/op | 7,284 ns/op | 9x |
| 20 | 145,864 ns/op | 14,045 ns/op | 10x |
No behavioral changes in either commit.
## Problem
The chat listing endpoint (`GetChatsByOwnerID`) was using
`fetchWithPostFilter`, which fetches N rows from the database and then
filters them in Go memory using RBAC checks. This causes a pagination
bug: if the user requests `limit=25` but some rows fail the auth check,
fewer than 25 rows are returned even though more authorized rows exist
in the database. The client may incorrectly assume it has reached the
end of the list.
## Solution
Switch to the same pattern used by `GetWorkspaces`, `GetTemplates`, and
`GetUsers`: `prepareSQLFilter` + `GetAuthorized*` variant. The RBAC
filter is compiled to a SQL WHERE clause and injected into the query
before `ORDER BY`/`LIMIT`, so the database returns exactly the requested
number of authorized rows.
Additionally, `GetChatsByOwnerID` is renamed to `GetChats` with
`OwnerID` as an optional (nullable) filter parameter, matching the
`GetWorkspaces` naming convention.
## Changes
| File | Change |
|------|--------|
| `queries/chats.sql` | Renamed to `GetChats`, `owner_id` now optional
via CASE/NULL, added `-- @authorize_filter` |
| `queries.sql.go` | Renamed constant, params struct (`GetChatsParams`),
and method |
| `querier.go` | Interface method renamed |
| `modelqueries.go` | Added `chatQuerier` interface +
`GetAuthorizedChats` impl |
| `dbauthz/dbauthz.go` | `GetChats` now uses `prepareSQLFilter` instead
of `fetchWithPostFilter` |
| `dbauthz/dbauthz_test.go` | Updated tests for SQL filter pattern |
| `dbmock/dbmock.go` | Renamed + added mock for `GetAuthorizedChats` |
| `dbmetrics/querymetrics.go` | Renamed + added metrics wrapper |
| `rbac/regosql/configs.go` | Added `ChatConverter` (maps `org_owner` to
empty string literal since `chats` has no `organization_id` column) |
| `rbac/authz.go` | Added `ConfigChats()` |
| `chats.go` | Handler uses renamed method with `uuid.NullUUID` |
| `searchquery/search.go` | Updated return type |
| `gitsync/worker.go` | Updated interface and call site |
| Various test files | Updated for renamed types |
## Summary
This PR removes two redundant chat rereads in `chatd`.
### Archive / unarchive
- `archiveChat` and `unarchiveChat` already come through
`httpmw.ChatParam`, so the handlers already have the `database.Chat`
row.
- Pass that row into `chatd.ArchiveChat` / `chatd.UnarchiveChat` instead
of rereading by ID before publishing the sidebar events.
### End-of-turn cleanup
- `processChat` no longer calls `GetChatByID` after the cleanup
transaction just to refresh the chat snapshot.
- Title generation already persists the generated title and emits its
own `title_change` event.
- To preserve best-effort title freshness for the cleanup path, the
async title-generation goroutine stores the generated title in per-turn
shared state and cleanup overlays it if available before publishing the
`status_change` event and dispatching push notifications.
## Why
- removes one DB read from archive / unarchive requests
- removes one DB read from completed turns, which is the larger hot-path
win
- keeps the existing pubsub/event contract intact instead of broadening
this into a larger event-model redesign
## Notes
- `title_change` remains the authoritative title update for clients
- cleanup does not wait for title generation; it uses the generated
title only when it is already available
## Summary
- reuse workspace agent context within a single `runChat()` turn
- remove duplicate latest-build agent lookups between
`resolveInstructions()` and `getWorkspaceConn()`
- avoid the extra `GetWorkspaceAgentByID` fetch when the selected
`WorkspaceAgent` already has the needed metadata
- add focused internal tests for reuse and refresh-on-dial-failure
## Why
This came out of a 5000-chat / 10-turn scaletest on bravo against a
single workspace.
The run completed successfully, but coderd stayed DB-pool bound, and one
workspace-backed hot path stood out:
- `GetWorkspaceAgentsInLatestBuildByWorkspaceID ≈ 46.7k`
- `GetWorkspaceByID ≈ 48.0k`
- `GetWorkspaceAgentByID ≈ 2.2k`
Within one `runChat()` turn, chatd was rediscovering the same workspace
agent multiple times just to resolve instructions and open the workspace
connection.
## What this changes
This PR introduces a **turn-local** workspace context helper so a single
acquired turn can:
- resolve the selected workspace agent once
- reuse that agent for instruction resolution
- reuse the same `AgentConn` for workspace tools and reload/compaction
This stays turn-local only, so a later turn on another replica still
rebuilds fresh context from the DB.
## Expected impact
This is an incremental improvement, not a full fix.
It should reduce duplicated workspace-agent lookups and shave some DB
pressure from a hot path for workspace-backed chats, while preserving
multi-replica correctness.
## Testing
- `go test ./coderd/chatd/...`
- `golangci-lint run ./coderd/chatd/...`
## Background
A 5000-chat scaletest (~50k turns, ~2m45s wall time) completed
successfully,
but the main bottleneck was **DB pool starvation from repeated reads**,
not
individually expensive SQL. The push/webpush path showed a few
especially noisy
reads:
- `GetLastChatMessageByRole` for push body generation
- `GetEnabledChatProviders` + `GetChatModelConfigByID` for push summary
model
resolution
- `GetWebpushSubscriptionsByUserID` for every webpush dispatch
This PR keeps the optimizations that remove those duplicate reads while
leaving
stream behavior unchanged.
## What changes in this PR
### 1. Reuse resolved chat state for push notifications
`maybeSendPushNotification` used to re-read the last assistant message
and
re-resolve the chat model/provider after `runChat` had already done that
work.
Now `runChat` returns the final assistant text plus the already-resolved
model
and provider keys, and the push goroutine uses that state directly.
That removes the extra push-path reads for:
- `GetLastChatMessageByRole`
- the second `resolveChatModel` path
- the provider/model lookups that came with that second resolution
### 2. Cache webpush subscriptions during dispatch
`Dispatch()` previously hit `GetWebpushSubscriptionsByUserID` on every
push. A
small per-user in-memory cache now avoids those repeated reads.
The follow-up fix keeps that optimization correct: `InvalidateUser()`
bumps a
per-user generation so an older in-flight fetch cannot repopulate the
cache with
pre-mutation data after subscribe/unsubscribe.
That preserves the cache win without letting local subscription changes
be
silently overwritten by stale fetch results.
## Why this is safe
- The push change only reuses data already produced during the same chat
run. It
does not change notification semantics; if there is no assistant text to
summarize, the existing fallback body still applies.
- The webpush change keeps the existing TTL and `410 Gone` cleanup
behavior. The
generation guard only prevents stale in-flight fetches from poisoning
the
shared cache after invalidation.
- The final PR does **not** change stream setup, pubsub/relay behavior,
or chat
status snapshot timing.
## Deliberately not included
- No stream-path optimization in `Subscribe`.
- No inline pubsub message payloads.
- No distributed cross-replica webpush cache invalidation.
## Problem
Models frequently use shell `&` instead of `run_in_background=true` when
starting long-running processes through `/agents`, causing them to die
shortly after starting. This happens because:
1. **No guidance in tool schema** — The `ExecuteArgs` struct had zero
`description` tags. The model saw `run_in_background: boolean
(optional)` with no explanation of when/why to use it.
2. **Shell `&` is silently broken** — `sh -c "command &"` forks the
process, the shell exits immediately, and the forked child becomes an
orphan not tracked by the process manager.
3. **No process group isolation** — The SSH subsystem sets `Setsid:
true` on spawned processes, but the agent process manager set no
`SysProcAttr` at all. Signals only hit the top-level `sh`, not child
processes.
## Investigation
Compared our implementation against **openai/codex** and **coder/mux**:
| Aspect | codex | mux | coder/coder (before) |
|--------|-------|-----|---------------------|
| Background flag | Yield/resume with `session_id` | `run_in_background`
with rich description | `run_in_background` with **no description** |
| `&` handling | `setsid()` + `killpg()` | `detached: true` +
`killProcessTree()` | **Nothing** — orphaned children escape |
| Process isolation | `setsid()` on every spawn | `set -m; nohup ...
setsid` for background | **No `SysProcAttr` at all** |
| Signal delivery | `killpg(pgid, sig)` — entire group | `kill -15
-\$pid` — negative PID | `proc.cmd.Process.Signal()` — **PID only** |
## Changes
### Fix 1: Add descriptions to `ExecuteArgs` (highest impact)
The model now sees explicit guidance: *"Use for long-running processes
like dev servers, file watchers, or builds. Do NOT use shell & — it will
not work correctly."*
### Fix 2: Update tool description
The top-level execute tool description now reinforces: *"Use
run_in_background=true for long-running processes. Never use shell '&'
for backgrounding."*
### Fix 3: Detect trailing `&` and auto-promote to background
Defense-in-depth: if the model still uses `command &`, we strip the `&`
and promote to `run_in_background=true` automatically. Correctly
distinguishes `&` from `&&`.
### Fix 4: Process group isolation (`Setpgid`)
New platform-specific files (`proc_other.go` / `proc_windows.go`)
following the same pattern as `agentssh/exec_other.go`. Every spawned
process gets its own process group.
### Fix 5: Process group signaling
`signal()` now uses `syscall.Kill(-pid, sig)` on Unix to signal the
entire process group, ensuring child processes from shell pipelines are
also cleaned up.
## Testing
All existing `agent/agentproc` tests pass. Both packages compile
cleanly.
Adds cursor-based pagination to the chat messages endpoint.
## Backend
- New `GetChatMessagesByChatIDPaginated` SQL query: returns messages in
`id DESC` order with a `before_id` keyset cursor and configurable
`limit`
- Handler parses `?before_id=N&limit=N` query params, uses the `LIMIT
N+1` trick to set `has_more` without a separate COUNT query
- Queued messages only returned on the first page (no cursor) since
they're always the most recent
- SDK client updated with `ChatMessagesPaginationOptions`
- Fully backward compatible: omitting params returns the 50 newest
messages
## Frontend
- Switches `getChatMessages` from `useQuery` to `useInfiniteQuery` with
cursor chaining via `getNextPageParam`
- Pages flattened and sorted by `id` ascending for chronological display
- `MessagesPaginationSentinel` component uses `IntersectionObserver`
(200px rootMargin prefetch) inside the existing `flex-col-reverse`
scroll container
- `flex-col-reverse` handles scroll anchoring natively when older
messages are prepended — no manual `scrollTop` adjustment needed (same
pattern as coder/blink)
## Why cursor-based instead of offset/limit
Offset-based pagination breaks when new messages arrive while paginating
backward (offsets shift, causing duplicates or missed messages). The
`before_id` cursor is stable regardless of inserts — each page is
deterministic.
## Problem
The sidebar diff status (PR icon, +additions/-deletions, file count) was
not updating in real-time. Users had to reload the page to see changes.
Two root causes:
1. **Frontend**: The `diff_status_change` WebSocket handler in
`AgentsPage.tsx` had an early `return` (line 398) that skipped
`updateInfiniteChatsCache`, so the sidebar's cache was never updated.
Even for other event types, the cache merge only spread `status` and
`title` — never `diff_status`.
2. **Server**: `publishChatPubsubEvent` in `chatd.go` constructed a
minimal `Chat` payload without `DiffStatus`, so even if the frontend
consumed the event, `updatedChat.diff_status` would be `undefined`.
## Fix
### Server (`coderd/chatd/chatd.go`)
- `publishChatPubsubEvent` now accepts an optional
`*codersdk.ChatDiffStatus` parameter; when non-nil it's set on the
outgoing `Chat` payload.
- `PublishDiffStatusChange` fetches the diff status from the DB,
converts it, and passes it through.
- Added `convertDBChatDiffStatus` (mirrors `coderd/chats.go`'s converter
to avoid circular import).
- All other callers pass `nil`.
### Frontend (`site/src/pages/AgentsPage/AgentsPage.tsx`)
- Removed the early `return` so `diff_status_change` events fall through
to the cache update logic.
- Added `isDiffStatusEvent` flag and spread `diff_status` into both the
infinite chats cache (sidebar) and the individual chat cache.
- Use t.Errorf in chattest non-streaming helpers so encoding
failures fail the test
- Thread testing.TB into writeResponsesAPIStreaming and log
SSE write errors instead of silently dropping them
- Bump createworkspace DB error log from Warn to Error
- Use errors.Join for timeout + output error in execute.go
Implement the backend for the desktop feature for agents.
- Adds a new `/api/experimental/chats/$id/desktop` endpoint to coderd
which exposes a VNC stream from a
[portabledesktop](https://github.com/coder/portabledesktop) process
running inside the workspace
- Adds a new `spawn_computer_use_agent` tool to chatd, which spawns a
subagent that has access to the `computer` tool which lets it interact
with the `portabledesktop` process running inside the workspace
- Adds the plumbing to make the above possible
There's a follow up frontend PR here:
https://github.com/coder/coder/pull/23006
Handle previously ignored error return values in coderd:
- coderd/chats.go: check sendEvent errors, log on failure
- coderd/chatd/chattest: thread testing.TB through server structs,
replace log.Printf with t.Logf, check writeSSEEvent errors
- coderd/chatd/chattool/createworkspace.go: log UpdateChatWorkspace
failure instead of discarding both return values
- coderd/chatd/chattool/execute.go: surface ProcessOutput error in
the timeout message returned to the caller
- coderd/provisionerdserver: log stream.Send failure in the
DownloadFile error helper
Add cost tracking for LLM chat interactions with microdollar precision.
## Changes
- Add `chatcost` package for per-message cost calculation using
`shopspring/decimal` for intermediate arithmetic
- **Ceil rounding policy**: fractional micros round UP to next whole
micro (applied once after summing all components)
- Database migration: `total_cost_micros` BIGINT column with historical
backfill and `created_at` index
- API endpoints: per-user cost summary and admin rollup under
`/api/experimental/chats/cost/`
- SDK types: `ChatCostSummary`, `ChatCostModelBreakdown`,
`ChatCostUserRollup`
- Fix `modeloptionsgen` to handle `decimal.Decimal` as opaque numeric
type
- Update frontend pricing test fixtures for string decimal types
## Design decisions
- `NULL` = unpriced (no matching model config), `0` = free
- Reasoning tokens included in output tokens (no double-counting)
- Integer microdollars (BIGINT) for storage and API responses
- Price config uses `decimal.Decimal` for exact parsing; totals use
`int64`
Frontend: #23037
Migration 000434 converts chat_messages.role from text to a Postgres
enum, rebuilds the partial index, and adds content_version smallint.
The column is backfilled with DEFAULT 0, then the default is dropped
so future inserts must set it explicitly.
Version 0 uses the role-aware heuristic from #22958. Version 1 (all
new inserts) stores []ChatMessagePart JSON for all roles, including
system messages. ParseContent takes database.ChatMessage directly
and dispatches on version internally. Unknown versions error.
All string(codersdk.ChatMessageRole*) casts at DB write sites are
replaced with database.ChatMessageRole* constants from sqlc.
Refs #22958
File-reference parts in user messages were flattened to `TextContent` at
write time because fantasy has no file-reference content type. The
frontend never saw them as structured parts.
This moves all write paths (user, assistant, tool) from fantasy envelope
format to `codersdk.ChatMessagePart`. The streaming layer (`chatloop`)
is untouched, the conversion happens at the serialization boundary in
`persistStep`.
Old rows are still readable. `ParseContent` uses a structural heuristic
(`isFantasyEnvelopeFormat`) to distinguish legacy envelopes from SDK
parts. We chose this over try/fallback because fantasy envelopes
partially unmarshal into `ChatMessagePart` (the `type` field matches)
while silently losing content. A guard test enforces that no SDK part
can produce the envelope shape.
This is forward-only: new rows are unreadable by old code. Chat is
behind a feature flag so rollback risk is contained.
Also adds a typed `ChatMessageRole` to replace raw strings and
`fantasy.MessageRole*` casts at the persistence boundary. The type
covers `ChatMessage.Role`, `ChatStreamMessagePart.Role`, the
`PublishMessagePart` callback chain, and all DB write sites.
`fantasy.MessageRole*` remains only where we build `fantasy.Message`
structs for LLM dispatch.
Separately, `ProviderMetadata` was leaking to SSE clients via
`publishMessagePart`. `StripInternal` now runs on both the SSE and REST
paths, covering this.
Other cleanup:
- Old `db2sdk.contentBlockToPart` silently dropped metadata on
text/reasoning/tool-call content. New code preserves it.
- `providerMetadataToOptions` now logs warnings instead of silently
returning nil.
- `db2sdk` shrinks from ~250 lines of parallel conversion to ~15 lines
delegating to `chatprompt.ParseContent()`, removing the `fantasy` import
entirely.
Refs #22821
## Summary
Moves the messages response out of `GET /chats/{id}` and into a
dedicated `GET /chats/{id}/messages` endpoint.
### Backend
- `GET /chats/{id}` now returns just the `Chat` object (no messages)
- `GET /chats/{id}/messages` is a new endpoint returning
`ChatMessagesResponse` with `messages` and `queued_messages`
- Added `ChatMessagesResponse` SDK type and `GetChatMessages` client
method
### Frontend
- `getChat()` API method returns `Chat` instead of `ChatWithMessages`
- Added `getChatMessages()` API method for the new endpoint
- Split `chatQuery` into two: `chatQuery` (metadata) and
`chatMessagesQuery` (messages)
- Updated all cache mutations, optimistic updates, and websocket
handlers
- Updated tests and stories
### Files changed
| File | Change |
|---|---|
| `coderd/coderd.go` | Register `GET /messages` route |
| `coderd/chats.go` | Simplify `getChat`, add `getChatMessages` handler
|
| `codersdk/chats.go` | New type + method, update `GetChat` return |
| `site/src/api/api.ts` | New method, update `getChat` |
| `site/src/api/queries/chats.ts` | New query, update cache mutations |
| `site/src/pages/AgentsPage/AgentDetail.tsx` | Use separate queries |
| `site/src/pages/AgentsPage/AgentDetail/ChatContext.ts` | Update types
and cache writes |
| `site/src/pages/AgentsPage/AgentsPage.tsx` | Update websocket cache
handler |
## Problem
When a chat is interrupted while tools are executing, the step content
(text, reasoning, tool calls, and partial tool results) was being lost.
Two gaps existed:
1. **During tool execution**: `executeTools` returns with error results
for interrupted tools, but the subsequent `PersistStep(ctx, ...)` fails
on the canceled context and returns `ErrInterrupted` without persisting
anything.
2. **PersistStep race**: If the context is canceled between the
post-tool interrupt check and the `PersistStep` call, the same loss
occurs.
This is inconsistent with how we handle stream interruptions (which
properly flush and persist partial content via `persistInterruptedStep`)
and how [coder/blink](https://github.com/coder/blink) handles
interruptions (always inserting the response message regardless of
execution phase).
## Fix
Two changes in `chatloop.go`:
- **Post-tool-execution interrupt check**: After `executeTools` returns,
check if the context was interrupted and route through
`persistInterruptedStep` (which uses `context.WithoutCancel` internally)
to save the accumulated content.
- **PersistStep fallback**: If `PersistStep` returns `ErrInterrupted`,
retry via `persistInterruptedStep` so partial content is not lost.
## Tests
- `TestRun_InterruptedDuringToolExecutionPersistsStep`: Verifies that
when a tool is blocked and the chat is interrupted, the step (text +
reasoning + tool call + tool error result) is persisted via the
interrupt-safe path.
- `TestRun_PersistStepInterruptedFallback`: Verifies that when
`PersistStep` itself returns `ErrInterrupted`, the step is retried via
the fallback path and content is saved.
## Problem
When a step contains both provider-executed tool calls (e.g. Anthropic
web search) and local tool calls in parallel, the next loop iteration
fails with the Anthropic API claiming the regular tool call has no
result. However, sending a new user message (which reloads messages from
the DB) works fine.
## Root cause
`toResponseMessages` was placing **all** tool results into the tool-role
message, regardless of `ProviderExecuted`. When Fantasy's Anthropic
provider later converted these messages for the API, it moved the
provider tool result from the tool message to the **end** of the
previous assistant message (`prevMsg.Content = append(...)`). This
placed `web_search_tool_result` **after** the regular `tool_use` block:
```
assistant: [server_tool_use(A), tool_use(B), web_search_tool_result(A)] ← wrong order
user: [tool_result(B)]
```
The persistence layer in `chatd.go` already handles this correctly —
provider-executed tool results stay in the assistant message, producing
the expected ordering:
```
assistant: [server_tool_use(A), web_search_tool_result(A), tool_use(B)] ← correct order
user: [tool_result(B)]
```
This is why reloading from the DB fixed it.
## Fix
In the `ContentTypeToolResult` case of `toResponseMessages`, route
provider-executed results to `assistantParts` instead of `toolParts`,
matching the persistence layer's behavior.
## Testing
Added
`TestToResponseMessages_ProviderExecutedToolResultInAssistantMessage`
which verifies that mixed provider+local tool results are split
correctly between the assistant and tool messages.
Mathias Fredriksson