Fixes all our Go file imports to match the preferred spec that we've _mostly_ been using. For example:
```
import (
"context"
"time"
"github.com/prometheus/client_golang/prometheus"
"golang.org/x/xerrors"
"gopkg.in/natefinch/lumberjack.v2"
"cdr.dev/slog/v3"
"github.com/coder/coder/v2/codersdk/agentsdk"
"github.com/coder/serpent"
)
```
3 groups: standard library, 3rd partly libs, Coder libs.
This PR makes the change across the codebase. The PR in the stack above modifies our formatting to maintain this state of affairs, and is a separate PR so it's possible to review that one in detail.
Upgrades to slog v3 which includes a small, but backward incompatible API change to the acceptible call arguments when logging. This change allows us to verify via compile time type checking that arguments are correct and won't cause a panic, as was possible in slog v1, which this replaces (v2 was tagged but never used in coder/coder).
It also updates dependencies that also use slog and were updated.
I've left the `aibridge` dependency as a commit SHA, under the assumption that the team there (cc @pawbana @dannykopping ) will tag and update the dependency soon and on their own schedule.
Other dependencies, I pushed new tags.
Refactors the CLI to create the `*codersdk.Client` in the handlers. This is groundwork for changing the `rootCmd.InitClient()` to use the new `ClientOption`s.
It also improves variable locality, scoping the Client to the handler. This makes misuse less likely and reduces the memory allocations to just the command being executed, rather than allocating a Client for every command regardless of whether it is executed.
Fixes https://github.com/coder/internal/issues/907
We convert `workspacesdk.AgentConn` to an interface and generate a mock
for it. This allows writing `coderd` tests that rely on the agent's HTTP
api to not have to set up an entire tailnet networking stack.
This pull request introduces support for external workspace management, allowing users to register and manage workspaces that are provisioned and managed outside of the Coder.
* coder external-workspaces create - Creates a new external workspace (this command extends coder create)
* Example: coder external-workspaces create ext-workspace --template=externally-managed-workspace -y
* Checks if template has coder_external_agent resource before creating a workspace
* coder external-workspaces list - Lists all external workspaces
* coder external-workspaces agent-instructions <workspace name> <agent name> - Retrieves agent connection instruction
* Example: coder external-workspaces agent-instructions ext-workspace main --output=json
This change allows a devcontainer to be opened via the agent syntax,
`coder open vscode <workspace>.<agent>` and removes the `--container`
option to simplify the subcommand. Accessing the subagent will behave
similarly to how the `--container` option behaved.
Fixescoder/internal#748
If the local IP address is not explicitly set, previously we assumed 127.0.0.1 (that is, IPv4 only localhost). This PR adds support to opportunistically _also_ listen on IPv6 ::1.
Currently, importing `codersdk` just to interact with the API requires
importing tailscale, which causes builds to fail unless manually using
our fork.
This PR updates the coder port-forward command to periodically inform coderd that the workspace is being used:
- Adds workspaceusage.Tracker which periodically batch-updates workspace LastUsedAt
- Adds coderd endpoint to signal workspace usage
- Updates coder port-forward to periodically hit this endpoint
- Modifies BatchUpdateWorkspacesLastUsedAt to avoid overwriting with stale data
Co-authored-by: Danny Kopping <danny@coder.com>
Fixes#10760
The coder CLI quietly accepts any subcommand arguments and silently swallows them.
Currently:
```sh
❯ coder | head -n5
coder v2.3.3+e491217
USAGE:
coder [global-flags] <subcommand>
```
```sh
❯ coder idontexist | head -n5
coder v2.3.3+e491217
USAGE:
coder [global-flags] <subcommand>
```
Now help output will not be show when there is an unknown subcommand error. Instead users will be given the command for the help output.
```sh
❯ coder idontexist
Encountered an error running "coder", see "coder --help" for more information
error: unrecognized subcommand "idontexist"
```
```sh
❯ coder iexistbut idontexist
Encountered an error running "coder iexistbut", see "coder iexistbut --help" for more information
error: unrecognized subcommand "idontexist"
```
Also this stuff: `Encountered an error running "coder iexistbut"... ` gets written to `os.Stdout` in `prettyErrorFormatter{w: os.Stderr, verbose: r.verbose}`, not sure how to test that output.
Fixes#10979
Testing code that listens on a specific port has created a long battle with flakes. Previous attempts to deal with this include opening a listener on a port chosen by the OS, then closing the listener, noting the port and starting the test with that port.
This still flakes, notably in macOS which has a proclivity to reuse ports quickly.
Instead of fighting with the chaos that is an OS networking stack, this PR fakes the host networking in tests.
I've taken a small step here, only faking out the Listen() calls that port-forward makes, but I think over time we should be transitioning all networking the CLI does to an abstract interface so we can fake it. This allows us to run in parallel without flakes and
presents an opportunity to test error paths as well.
Adds a Logger to cli Invocation and standardizes CLI commands to use it. clitest creates a test logger by default so that CLI command logs are captured in the test logs.
CLI commands that do their own log configuration are modified to add sinks to the existing logger, rather than create a new one. This ensures we still capture logs in CLI tests.
I've said it before, I'll say it again: you can't create a timed context before calling `t.Parallel()` and then use it after.
Fixes flakes like https://github.com/coder/coder/actions/runs/6716682414/job/18253279157
I've chosen just to drop `t.Parallel()` entirely rather than create a second context after the parallel call, since the vast majority of the test time happens before where the parallel call was. It does all the tailnet setup before `t.Parallel()`.
Leaving a call to `t.Parallel()` is a bug risk for future maintainers to come in and use the wrong context in the latter part of the test by accident.
* chore: add /v2 to import module path
go mod requires semantic versioning with versions greater than 1.x
This was a mechanical update by running:
```
go install github.com/marwan-at-work/mod/cmd/mod@latest
mod upgrade
```
Migrate generated files to import /v2
* Fix gen
* chore: rename `AgentConn` to `WorkspaceAgentConn`
The codersdk was becoming bloated with consts for the workspace
agent that made no sense to a reader. `Tailnet*` is an example
of these consts.
* chore: remove `Get` prefix from *Client functions
* chore: remove `BypassRatelimits` option in `codersdk.Client`
It feels wrong to have this as a direct option because it's so infrequently
needed by API callers. It's better to directly modify headers in the two
places that we actually use it.
* Merge `appearance.go` and `buildinfo.go` into `deployment.go`
* Merge `experiments.go` and `features.go` into `deployment.go`
* Fix `make gen` referencing old type names
* Merge `error.go` into `client.go`
`codersdk.Response` lived in `error.go`, which is wrong.
* chore: refactor workspace agent functions into agentsdk
It was odd conflating the codersdk that clients should use
with functions that only the agent should use. This separates
them into two SDKs that are closely coupled, but separate.
* Merge `insights.go` into `deployment.go`
* Merge `organizationmember.go` into `organizations.go`
* Merge `quota.go` into `workspaces.go`
* Rename `sse.go` to `serversentevents.go`
* Rename `codersdk.WorkspaceAppHostResponse` to `codersdk.AppHostResponse`
* Format `.vscode/settings.json`
* Fix outdated naming in `api.ts`
* Fix app host response
* Fix unsupported type
* Fix imported type
* fix: Improve tailnet connections by reducing timeouts
This awaits connection ping before running a dial. Before,
we were hitting the TCP retransmission and handshake timeouts,
which could intermittently add 1 or 5 seconds to a connection
being initialized.
* Update Tailscale
* feat: Add bufferring to provisioner job logs
This should improve overall build performance, and especially under load.
It removes the old `id` column on the `provisioner_job_logs` table
and replaces it with an auto-incrementing big integer to preserve order.
Funny enough, we never had to care about order before because inserts
would at minimum be 1ms different. Now they aren't, so the order needs
to be preserved.
* Fix log bufferring
* Fix frontend log streaming
* Fix JS test
* feat: HA tailnet coordinator
* fixup! feat: HA tailnet coordinator
* fixup! feat: HA tailnet coordinator
* remove printlns
* close all connections on coordinator
* impelement high availability feature
* fixup! impelement high availability feature
* fixup! impelement high availability feature
* fixup! impelement high availability feature
* fixup! impelement high availability feature
* Add replicas
* Add DERP meshing to arbitrary addresses
* Move packages to highavailability folder
* Move coordinator to high availability package
* Add flags for HA
* Rename to replicasync
* Denest packages for replicas
* Add test for multiple replicas
* Fix coordination test
* Add HA to the helm chart
* Rename function pointer
* Add warnings for HA
* Add the ability to block endpoints
* Add flag to disable P2P connections
* Wow, I made the tests pass
* Add replicas endpoint
* Ensure close kills replica
* Update sql
* Add database latency to high availability
* Pipe TLS to DERP mesh
* Fix DERP mesh with TLS
* Add tests for TLS
* Fix replica sync TLS
* Fix RootCA for replica meshing
* Remove ID from replicasync
* Fix getting certificates for meshing
* Remove excessive locking
* Fix linting
* Store mesh key in the database
* Fix replica key for tests
* Fix types gen
* Fix unlocking unlocked
* Fix race in tests
* Update enterprise/derpmesh/derpmesh.go
Co-authored-by: Colin Adler <colin1adler@gmail.com>
* Rename to syncReplicas
* Reuse http client
* Delete old replicas on a CRON
* Fix race condition in connection tests
* Fix linting
* Fix nil type
* Move pubsub to in-memory for twenty test
* Add comment for configuration tweaking
* Fix leak with transport
* Fix close leak in derpmesh
* Fix race when creating server
* Remove handler update
* Skip test on Windows
* Fix DERP mesh test
* Wrap HTTP handler replacement in mutex
* Fix error message for relay
* Fix API handler for normal tests
* Fix speedtest
* Fix replica resend
* Fix derpmesh send
* Ping async
* Increase wait time of template version jobd
* Fix race when closing replica sync
* Add name to client
* Log the derpmap being used
* Don't connect if DERP is empty
* Improve agent coordinator logging
* Fix lock in coordinator
* Fix relay addr
* Fix race when updating durations
* Fix client publish race
* Run pubsub loop in a queue
* Store agent nodes in order
* Fix coordinator locking
* Check for closed pipe
Co-authored-by: Colin Adler <colin1adler@gmail.com>
* feat: Use Tailscale networking by default
Removal of WebRTC code will happen in another PR, but it
felt dangerious to default and remove in a single commit.
Ideally, we can release this version and collect final
thoughts and feedback before a full commitment.
* Remove UNIX forwarding
Tailscale doesn't support this, and adding support
for it shouldn't block our rollout. Customers can
always forward over SSH.
* Update cli/portforward_test.go
Co-authored-by: Dean Sheather <dean@deansheather.com>
Co-authored-by: Dean Sheather <dean@deansheather.com>
* fix: Add coder user to docker group on installation
This makes for a simpler setup, and reduces the likelihood
a user runs into a strange issue.
* Add wgnet
* Add ping
* Add listening
* Finish refactor to make this work
* Add interface for swapping
* Fix conncache with interface
* chore: update gvisor
* fix tailscale types
* linting
* more linting
* Add coordinator
* Add coordinator tests
* Fix coordination
* It compiles!
* Move all connection negotiation in-memory
* Migrate coordinator to use net.conn
* Add closed func
* Fix close listener func
* Make reconnecting PTY work
* Fix reconnecting PTY
* Update CI to Go 1.19
* Add CLI flags for DERP mapping
* Fix Tailnet test
* Rename ConnCoordinator to TailnetCoordinator
* Remove print statement from workspace agent test
* Refactor wsconncache to use tailnet
* Remove STUN from unit tests
* Add migrate back to dump
* chore: Upgrade to Go 1.19
This is required as part of #3505.
* Fix reconnecting PTY tests
* fix: update wireguard-go to fix devtunnel
* fix migration numbers
* linting
* Return early for status if endpoints are empty
* Update cli/server.go
Co-authored-by: Colin Adler <colin1adler@gmail.com>
* Update cli/server.go
Co-authored-by: Colin Adler <colin1adler@gmail.com>
* Fix frontend entites
* Fix agent bicopy
* Fix race condition for the last node
* Fix down migration
* Fix connection RBAC
* Fix migration numbers
* Fix forwarding TCP to a local port
* Implement ping for tailnet
* Rename to ForceHTTP
* Add external derpmapping
* Expose DERP region names to the API
* Add global option to enable Tailscale networking for web
* Mark DERP flags hidden while testing
* Update DERP map on reconnect
* Add close func to workspace agents
* Fix race condition in upstream dependency
* Fix feature columns race condition
Co-authored-by: Colin Adler <colin1adler@gmail.com>
* fix: Improve shutdown procedure of ssh, portforward, wgtunnel cmds
We could turn it into a practice to wrap `cmd.Context()` so that we have
more fine-grained control of cancellation. Sometimes in tests we may be
running commands with a context that is never canceled.
Related to #3221
* fix: Set ssh session stderr to stderr
Spike Curtis