Summary:
There's no reason to build in `test/js`, since we have e2e tests that build.
Details:
- Remove superfluous `yarn build` from `test/js` step in CI
Relates to #1004 but does not fix it.
* Improve CLI documentation
* feat: Allow workspace resources to attach multiple agents
This enables a "kubernetes_pod" to attach multiple agents that
could be for multiple services. Each agent is required to have
a unique name, so SSH syntax is:
`coder ssh <workspace>.<agent>`
A resource can have zero agents too, they aren't required.
* Add tree view
* Improve table UI
* feat: Allow workspace resources to attach multiple agents
This enables a "kubernetes_pod" to attach multiple agents that
could be for multiple services. Each agent is required to have
a unique name, so SSH syntax is:
`coder ssh <workspace>.<agent>`
A resource can have zero agents too, they aren't required.
* Rename `tunnel` to `skip-tunnel`
This command was `true` by default, which causes
a confusing user experience.
* Add disclaimer about editing templates
* Add help to template create
* Improve workspace create flow
* Add end-to-end test for config-ssh
* Improve testing of config-ssh
* Fix workspace list
* feat: Add support for VS Code and JetBrains Gateway via SSH
This fixes various bugs that made this not work:
- Incorrect max message size in `peer`
- Incorrect reader buffer size in `peer`
- Lack of SFTP support in `agent`
- Lack of direct-tcpip support in `agent`
- Misuse of command from session. It should always use the shell
- Blocking on SSH session, only allowing one at a time
Fixes#833 too.
* Fix config-ssh command with socat
* ci: Enable forks to run CI
All steps that require tokens are optional for forks,
and will be skipped if the owner is not "coder".
* Empty commit to force CI
Resolves: #444
Summary:
This commit installs and configures a GH action for chromatic. Chromatic
is used for snapshot testing build-over-build.
Details:
* chore: install chromatic
* chore: add chromatic package.json script
Suggested by the docs for convenience so that we can run chromatic like:
```console
yarn run chromatic ...
```
* chore: gitignore storybook builds
* ci: configure chromatic
This action configures chromatic to run in CI on pushes to all branches.
By running this in CI, we get the following:
- snapshot (build-over-build)
- checks in our CI
The snapshots and build-over-build behavior are per branch; this way we
can work on a feature branch without worrying about changes being made
to mainline independently.
* chore: remove manual storybook build from CI
This is now the responsibility of Chromatic
* ci: Fix dogfood installation by forcing default configurations
The dpkg prompt to override config files was
appearing, but this will auto-approve it.
* Add CAP_NET_BIND_SERVICE to allow listening on :443
* feat: Add TLS support
This adds numerous flags with inspiration taken from Vault
for configuring TLS inside Coder.
This enables secure deployments without a proxy, like Cloudflare.
* Update cli/start.go
Co-authored-by: Colin Adler <colin@coder.com>
* Fix flag help in coder.env
Co-authored-by: Colin Adler <colin@coder.com>
* feat: Add systemd service and production deployment
This modifies CI to use a dpkg produced from release to update and
run Coder on a tiny VM in GCP.
It's intentionally kept simple, because customers should
be able to get this same easy install experience.
* Update globalSetup.ts
* Update globalSetup.ts
* Update globalSetup.ts
* Update coder.yaml
* Use pinned version of Go
* chore: Improve CI builds by caching Go modules
* Skip running with `race` on non-Linux systems
* Fix darwin file descriptor error
* Fix log after close
* Improve PostgreSQL test speeds
* Fix parallel connections with PostgreSQL tests
* Fix CI flake
* Separate test/go into PostgreSQL
This fixes#433 - a test flake in E2E (intermittent `ESOCKETTIMEDOUT` errors on MacOS).
The main issue is that, occasionally, for very large dependencies (like `@material-ui/icons`) - yarn can actually time out! We researched this in-depth in v1: https://github.com/coder/m/pull/10040 and fixed it successfully there, by increasing the timeout for yarn.
However, this also highlighted the fact that our `node_modules` caching behavior wasn't correct - we should very rarely see a timeout issue like this, because `@material-ui/icons` should be cached.
It turns out that we weren't falling back to the latest cached `node_modules` if there was a miss - so anytime the lock file changed, we'd invalidate the cache, and not restore the previous one. This can be improved by using the [`restore-keys`](https://github.com/coder/m/pull/10040) parameter of the [`@actions/cache`](https://github.com/actions/cache)... and in fact we already do this for the `go` dependencies.
So this fix does two things:
- Improve the caching behavior, such that we should rarely have to install `@material-ui/icons` (and other large dependencies)
- When we do have to install, update the timeout so that we can avoid random `ESOCKETTIMEDOUT` errors
Fix for #348 - migrate our NextJS project to a pure webpack project w/ a single bundle
- [x] Switch from `next/link` to `react-router-dom`'s link
> This part was easy - just change the import to `import { Link } from "react-router-dom"` and `<Link href={...} />` to `<Link to={...} />`
- [x] Switch from `next/router` to `react-router-dom`'s paradigms (`useNavigation`, `useLocation`, and `useParams`)
> `router.push` can be converted to `navigate(...)` (provided by the `useNavigate` hook)
> `router.replace` can be converted `navigate(..., {replace: true})`
> Query parameters (`const { query } = useRouter`) can be converted to `const query = useParams()`)
- [x] Implement client-side routing with `react-router-dom`
> Parameterized routes in NextJS like `projects/[organization]/[project]` would look like:
> ```
> <Route path="projects">
> <Route path=":organization/:project">
> <Route index element={<ProjectPage />} />
> </Route>
> </Route>
> ```
I've hooked up a `build:analyze` command that spins up a server to show the bundle size:
<img width="1303" alt="image" src="https://user-images.githubusercontent.com/88213859/157496889-87c5fdcd-fad1-4f2e-b7b6-437aebf99641.png">
The bundle looks OK, but there are some opportunities for improvement - the heavy-weight dependencies, like React, ReactDOM, Material-UI, and lodash could be brought in via a CDN: https://stackoverflow.com/questions/50645796/how-to-import-reactjs-material-ui-using-a-cdn-through-webpacks-externals
This brings an initial E2E test (really, an integration test - it's only running the server locally, as opposed to against a deployment - but it'd be easy to point playwright to a deployment).
Demo gif:
This test exercises a minimal flow for login:
- Run the `coderd` binary to start a server on 3000
- Create an initial user as part of setup
- Go through the login flow and verify we land on the projects page
It will be useful to have to ensure that #360 doesn't introduce a regression in the login flow
Future E2E tests that would be useful:
- Create a project & verify it shows in the UI
- Create a workspace and verify it shows in the UI
* fix: Leaking yamux session after HTTP handler is closed
Closes#317. The httptest server cancels the context after the connection
is closed, but if a connection takes a long time to close, the request
would never end. This applies a context to the entire listener that cancels
on test cleanup.
After discussion with @bryphe-coder, reducing the parallel limit on
Windows is likely to reduce failures as well.
* Switch to windows-2022 to improve decompression
* Invalidate cache on matrix OS
Fixes#244
- Adds `jest-junit`
- Configures `jest-junit` to output `junit.xml` in `site/test_results`
- Uploads the emitted `junit.xml` to datadog as part of the `test/js` workflow
This just adds a very simple dockerfile for deploying `coderd` (and later `provisionerd`).
This adds a `deploy` directory at the root, and a `make docker/build` command to the makefile.
Thanks @jawnsy for the all the help 😄
* ci: Replace DataDog CI with custom upload script
This will reduce CI time by ~6 minutes across all of
our runners. It's a bit janky, but I believe worth
the slight maintainance burden.
* Fix test race when job would complete too early
* Fix job cancelation override
* Fix race where provisioner job is inserted before project version
* chore: Rename ProjectHistory to ProjectVersion
Version more accurately represents version storage. This
forks from the WorkspaceHistory name, but I think it's
easier to understand Workspace history.
* Rename files
* Standardize tests a bit more
* Remove Server struct from coderdtest
* Improve test coverage for workspace history
* Fix linting errors
* Fix coderd test leak
* Fix coderd test leak
* Improve workspace history logs
* Standardize test structure for codersdk
* Fix linting errors
* Fix WebSocket compression
* Update coderd/workspaces.go
Co-authored-by: Bryan <bryan@coder.com>
* Add test for listing project parameters
* Cache npm dependencies with setup node
* Remove windows npm cache key
Co-authored-by: Bryan <bryan@coder.com>
This hooks up `storybook`, which the front-end team has enjoyed using in the v1 codebase - it makes it quick and easy to view and test components in isolation.
The `<LoadingButton />` has a simple story added now, so if you run `yarn storybook`, you can preview it in various states:
This will be helpful as we bring more front-end devs to help build v2 out.
* feat: Add history middleware parameters
These will be used for streaming logs, checking status,
and other operations related to workspace and project
history.
* refactor: Move all HTTP routes to top-level struct
Nesting all structs behind their respective structures
is leaky, and promotes naming conflicts between handlers.
Our HTTP routes cannot have conflicts, so neither should
function naming.
* Add provisioner daemon routes
* Add periodic updates
* Skip pubsub if short
* Return jobs with WorkspaceHistory
* Add endpoints for extracting singular history
* The full end-to-end operation works
* fix: Disable compression for websocket dRPC transport (#145)
There is a race condition in the interop between the websocket and `dRPC`: https://github.com/coder/coder/runs/5038545709?check_suite_focus=true#step:7:117 - it seems both the websocket and dRPC feel like they own the `byte[]` being sent between them. This can lead to data races, in which both `dRPC` and the websocket are writing.
This is just tracking some experimentation to fix that race condition
## Run results: ##
- Run 1: peer test failure
- Run 2: peer test failure
- Run 3: `TestWorkspaceHistory/CreateHistory` - https://github.com/coder/coder/runs/5040858460?check_suite_focus=true#step:8:45
```
status code 412: The provided project history is running. Wait for it to complete importing!`
```
- Run 4: `TestWorkspaceHistory/CreateHistory` - https://github.com/coder/coder/runs/5040957999?check_suite_focus=true#step:7:176
```
workspacehistory_test.go:122:
Error Trace: workspacehistory_test.go:122
Error: Condition never satisfied
Test: TestWorkspaceHistory/CreateHistory
```
- Run 5: peer failure
- Run 6: Pass ✅
- Run 7: Peer failure
## Open Questions: ##
### Is `dRPC` or `websocket` at fault for the data race?
It looks like this condition is specifically happening when `dRPC` decides to [`SendError`]). This constructs a new byte payload from [`MarshalError`](https://github.com/storj/drpc/blob/f6e369438f636b47ee788095d3fc13062ffbd019/drpcwire/error.go#L15) - so `dRPC` has created this buffer and owns it.
From `dRPC`'s perspective, the callstack looks like this:
- [`sendPacket`](https://github.com/storj/drpc/blob/f6e369438f636b47ee788095d3fc13062ffbd019/drpcstream/stream.go#L253)
- [`writeFrame`](https://github.com/storj/drpc/blob/f6e369438f636b47ee788095d3fc13062ffbd019/drpcwire/writer.go#L65)
- [`AppendFrame`](https://github.com/storj/drpc/blob/f6e369438f636b47ee788095d3fc13062ffbd019/drpcwire/packet.go#L128)
- with finally the data race happening here:
```go
// AppendFrame appends a marshaled form of the frame to the provided buffer.
func AppendFrame(buf []byte, fr Frame) []byte {
...
out := buf
out = append(out, control). // <---------
```
This should be fine, since `dPRC` create this buffer, and is taking the byte buffer constructed from `MarshalError` and tacking a bunch of headers on it to create a proper frame.
Once `dRPC` is done writing, it _hangs onto the buffer and resets it here__: https://github.com/storj/drpc/blob/f6e369438f636b47ee788095d3fc13062ffbd019/drpcwire/writer.go#L73
However... the websocket implementation, once it gets the buffer, it runs a `statelessDeflate` [here](https://github.com/nhooyr/websocket/blob/8dee580a7f74cf1713400307b4eee514b927870f/write.go#L180), which compresses the buffer on the fly. This functionality actually [mutates the buffer in place](https://github.com/klauspost/compress/blob/a1a9cfc821f00faf2f5231beaa96244344d50391/flate/stateless.go#L94), which is where get our race.
In the case where the `byte[]` aren't being manipulated anywhere else, this compress-in-place operation would be safe, and that's probably the case for most over-the-wire usages. In this case, though, where we're plumbing `dRPC` -> websocket, they both are manipulating it (`dRPC` is reusing the buffer for the next `write`, and `websocket` is compressing on the fly).
### Why does cloning on `Read` fail?
Get a bunch of errors like:
```
2022/02/02 19:26:10 [WARN] yamux: frame for missing stream: Vsn:0 Type:0 Flags:0 StreamID:0 Length:0
2022/02/02 19:26:25 [ERR] yamux: Failed to read header: unexpected EOF
2022/02/02 19:26:25 [ERR] yamux: Failed to read header: unexpected EOF
2022/02/02 19:26:25 [WARN] yamux: frame for missing stream: Vsn:0 Type:0 Flags:0 StreamID:0 Length:0
```
# UPDATE:
We decided we could disable websocket compression, which would avoid the race because the in-place `deflate` operaton would no longer be run. Trying that out now:
- Run 1: ✅
- Run 2: https://github.com/coder/coder/runs/5042645522?check_suite_focus=true#step:8:338
- Run 3: ✅
- Run 4: https://github.com/coder/coder/runs/5042988758?check_suite_focus=true#step:7:168
- Run 5: ✅
* fix: Remove race condition with acquiredJobDone channel (#148)
Found another data race while running the tests: https://github.com/coder/coder/runs/5044320845?check_suite_focus=true#step:7:83
__Issue:__ There is a race in the p.acquiredJobDone chan - in particular, there can be a case where we're waiting on the channel to finish (in close) with <-p.acquiredJobDone, but in parallel, an acquireJob could've been started, which would create a new channel for p.acquiredJobDone. There is a similar race in `close(..)`ing the channel, which also came up in test runs.
__Fix:__ Instead of recreating the channel everytime, we can use `sync.WaitGroup` to accomplish the same functionality - a semaphore to make close wait for the current job to wrap up.
* fix: Bump up workspace history timeout (#149)
This is an attempted fix for failures like: https://github.com/coder/coder/runs/5043435263?check_suite_focus=true#step:7:32
Looking at the timing of the test:
```
t.go:56: 2022-02-02 21:33:21.964 [DEBUG] (terraform-provisioner) <provision.go:139> ran apply
t.go:56: 2022-02-02 21:33:21.991 [DEBUG] (provisionerd) <provisionerd.go:162> skipping acquire; job is already running
t.go:56: 2022-02-02 21:33:22.050 [DEBUG] (provisionerd) <provisionerd.go:162> skipping acquire; job is already running
t.go:56: 2022-02-02 21:33:22.090 [DEBUG] (provisionerd) <provisionerd.go:162> skipping acquire; job is already running
t.go:56: 2022-02-02 21:33:22.140 [DEBUG] (provisionerd) <provisionerd.go:162> skipping acquire; job is already running
t.go:56: 2022-02-02 21:33:22.195 [DEBUG] (provisionerd) <provisionerd.go:162> skipping acquire; job is already running
t.go:56: 2022-02-02 21:33:22.240 [DEBUG] (provisionerd) <provisionerd.go:162> skipping acquire; job is already running
workspacehistory_test.go:122:
Error Trace: workspacehistory_test.go:122
Error: Condition never satisfied
Test: TestWorkspaceHistory/CreateHistory
```
It appears that the `terraform apply` job had just finished - with less than a second to spare until our `require.Eventually` completes - but there's still work to be done (ie, collecting the state files). So my suspicion is that terraform might, in some cases, exceed our 5s timeout.
Note that in the setup for this test - there is a similar project history wait that waits for 15s, so I borrowed that here.
In the future - we can look at potentially using a simple echo provider to exercise this in the unit test, in a way that is more reliable in terms of timing. I'll log an issue to track that.
Co-authored-by: Bryan <bryan@coder.com>
This refactors the front-end collateral to all live within `site` - so no `package.json` at the root.
The reason we had this initially is that the jest test run and NextJS actually require having _two_ different `tsconfig`s - Next needs `jsx:"preserve"`, while jest needs `jsx:"react"` - we were using `tsconfig`s at different levels at the hierarchy to manage this.
I changed this behavior to still use two different `tsconfig.json`s, which is mandatory - but just side-by-side in `site`.
Once that's fixed, it was easy to move everything into `site`
Follow up from: https://github.com/coder/coder/pull/118#discussion_r796244577
@kylecarbs and I were debugging a gnarly postgres issue over the weekend, and unfortunately it looks like it is still coming up occassionally: https://github.com/coder/coder/runs/5014420662?check_suite_focus=true#step:8:35 - so thought this might be a good testing Monday task.
Intermittently, the test would fail with something like a `401` - invalid e-mail, or a `409` - initial user already created. This was quite surprising, because the tests are designed to spin up their own, isolated database.
We tried a few things to debug this...
## Attempt 1: Log out the generated port numbers when running the docker image.
Based on the errors, it seemed like one test must be connecting to another test's database - that would explain why we'd get these conflicts! However, logging out the port number that came from docker always gave a unique number... and we couldn't find evidence of one database connecting to another.
## Attempt 2: Store the database in unique, temporary folder.
@kylecarbs and I found that the there was a [volume](https://github.com/docker-library/postgres/blob/a83005b407ee6d810413500d8a041c957fb10cf0/11/alpine/Dockerfile#L155) for the postgres data... so @kylecarbs implemented mounting the volume to a unique, per-test temporary folder in https://github.com/coder/coder/pull/89
It sounded really promising... but unfortunately we hit the issue again!
### Attempt 3... this PR
After we hit the failure again, we noticed in the `docker ps` logs something quite strange:
When the docker image is run - it creates two port bindings, an IPv4 and an IPv6 one. These _should be the same_ - but surprisingly, they can sometimes be different. It isn't deterministic, and seems to be more common when there are multiple containers running. Importantly, __they can overlap__ as in the above image.
Turns out, it seems this is a docker bug: https://github.com/moby/moby/issues/42442 - which may be fixed in newer versions.
To work around this bug, we have to manipulate the port bindings (like you would with `-p`) at the command line. We can do this with `docker`/`dockertest`, but it means we have to get a free port ahead of time to know which port to map.
With that fix in - the `docker ps` is a little more sane:
...and hopefully means we can safely run the containers in parallel again.
Use the native 'concurrency' configuration feature to cancel
concurrent builds, rather than the cancel-workflow-action.
This also allows us to reduce permissions for the workflow.
Garrett Delfosse
