docs: restructure docs (#14421)

Closes #13434 
Supersedes #14182

---------

Co-authored-by: Ethan <39577870+ethanndickson@users.noreply.github.com>
Co-authored-by: Ethan Dickson <ethan@coder.com>
Co-authored-by: Ben Potter <ben@coder.com>
Co-authored-by: Stephen Kirby <58410745+stirby@users.noreply.github.com>
Co-authored-by: Stephen Kirby <me@skirby.dev>
Co-authored-by: EdwardAngert <17991901+EdwardAngert@users.noreply.github.com>
Co-authored-by: Edward Angert <EdwardAngert@users.noreply.github.com>
This commit is contained in:
Muhammad Atif Ali
2024-10-05 08:52:04 -07:00
committed by GitHub
parent 288df75686
commit 419eba5fb6
298 changed files with 5009 additions and 3889 deletions
@@ -0,0 +1,74 @@
# High Availability
High Availability (HA) mode solves for horizontal scalability and automatic
failover within a single region. When in HA mode, Coder continues using a single
Postgres endpoint.
[GCP](https://cloud.google.com/sql/docs/postgres/high-availability),
[AWS](https://docs.aws.amazon.com/prescriptive-guidance/latest/saas-multitenant-managed-postgresql/availability.html),
and other cloud vendors offer fully-managed HA Postgres services that pair
nicely with Coder.
For Coder to operate correctly, Coderd instances should have low-latency
connections to each other so that they can effectively relay traffic between
users and workspaces no matter which Coderd instance users or workspaces connect
to. We make a best-effort attempt to warn the user when inter-Coderd latency is
too high, but if requests start dropping, this is one metric to investigate.
We also recommend that you deploy all Coderd instances such that they have
low-latency connections to Postgres. Coderd often makes several database
round-trips while processing a single API request, so prioritizing low-latency
between Coderd and Postgres is more important than low-latency between users and
Coderd.
Note that this latency requirement applies _only_ to Coder services. Coder will
operate correctly even with few seconds of latency on workspace <-> Coder and
user <-> Coder connections.
## Setup
Coder automatically enters HA mode when multiple instances simultaneously
connect to the same Postgres endpoint.
HA brings one configuration variable to set in each Coderd node:
`CODER_DERP_SERVER_RELAY_URL`. The HA nodes use these URLs to communicate with
each other. Inter-node communication is only required while using the embedded
relay (default). If you're using [custom relays](./index.md#custom-relays),
Coder ignores `CODER_DERP_SERVER_RELAY_URL` since Postgres is the sole
rendezvous for the Coder nodes.
`CODER_DERP_SERVER_RELAY_URL` will never be `CODER_ACCESS_URL` because
`CODER_ACCESS_URL` is a load balancer to all Coder nodes.
Here's an example 3-node network configuration setup:
| Name | `CODER_HTTP_ADDRESS` | `CODER_DERP_SERVER_RELAY_URL` | `CODER_ACCESS_URL` |
| --------- | -------------------- | ----------------------------- | ------------------------ |
| `coder-1` | `*:80` | `http://10.0.0.1:80` | `https://coder.big.corp` |
| `coder-2` | `*:80` | `http://10.0.0.2:80` | `https://coder.big.corp` |
| `coder-3` | `*:80` | `http://10.0.0.3:80` | `https://coder.big.corp` |
## Kubernetes
If you installed Coder via
[our Helm Chart](../../install/kubernetes.md#4-install-coder-with-helm), just
increase `coder.replicaCount` in `values.yaml`.
If you installed Coder into Kubernetes by some other means, insert the relay URL
via the environment like so:
```yaml
env:
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: CODER_DERP_SERVER_RELAY_URL
value: http://$(POD_IP)
```
Then, increase the number of pods.
## Up next
- [Read more on Coder's networking stack](./index.md)
- [Install on Kubernetes](../../install/kubernetes.md)
+199
View File
@@ -0,0 +1,199 @@
# Networking
Coder's network topology has three types of nodes: workspaces, coder servers,
and users.
The coder server must have an inbound address reachable by users and workspaces,
but otherwise, all topologies _just work_ with Coder.
When possible, we establish direct connections between users and workspaces.
Direct connections are as fast as connecting to the workspace outside of Coder.
When NAT traversal fails, connections are relayed through the coder server. All
user <-> workspace connections are end-to-end encrypted.
[Tailscale's open source](https://tailscale.com) backs our networking logic.
## Requirements
In order for clients and workspaces to be able to connect:
> **Note:** We strongly recommend that clients connect to Coder and their
> workspaces over a good quality, broadband network connection. The following
> are minimum requirements:
>
> - better than 400ms round-trip latency to the Coder server and to their
> workspace
> - better than 0.5% random packet loss
- All clients and agents must be able to establish a connection to the Coder
server (`CODER_ACCESS_URL`) over HTTP/HTTPS.
- Any reverse proxy or ingress between the Coder control plane and
clients/agents must support WebSockets.
> **Note:** We strongly recommend that clients connect to Coder and their
> workspaces over a good quality, broadband network connection. The following
> are minimum requirements:
>
> - better than 400ms round-trip latency to the Coder server and to their
> workspace
> - better than 0.5% random packet loss
In order for clients to be able to establish direct connections:
> **Note:** Direct connections via the web browser are not supported. To improve
> latency for browser-based applications running inside Coder workspaces in
> regions far from the Coder control plane, consider deploying one or more
> [workspace proxies](./workspace-proxies.md).
- The client is connecting using the CLI (e.g. `coder ssh` or
`coder port-forward`). Note that the
[VSCode extension](https://marketplace.visualstudio.com/items?itemName=coder.coder-remote)
and [JetBrains Plugin](https://plugins.jetbrains.com/plugin/19620-coder/), and
[`ssh coder.<workspace>`](../../reference/cli/config-ssh.md) all utilize the
CLI to establish a workspace connection.
- Either the client or workspace agent are able to discover a reachable
`ip:port` of their counterpart. If the agent and client are able to
communicate with each other using their locally assigned IP addresses, then a
direct connection can be established immediately. Otherwise, the client and
agent will contact
[the configured STUN servers](../../reference/cli/server.md#derp-server-stun-addresses)
to try and determine which `ip:port` can be used to communicate with their
counterpart. See [STUN and NAT](./stun.md) for more details on how this
process works.
- All outbound UDP traffic must be allowed for both the client and the agent on
**all ports** to each others' respective networks.
- To establish a direct connection, both agent and client use STUN. This
involves sending UDP packets outbound on `udp/3478` to the configured
[STUN server](../../reference/cli/server.md#--derp-server-stun-addresses).
If either the agent or the client are unable to send and receive UDP packets
to a STUN server, then direct connections will not be possible.
- Both agents and clients will then establish a
[WireGuard](https://www.wireguard.com/) tunnel and send UDP traffic on
ephemeral (high) ports. If a firewall between the client and the agent
blocks this UDP traffic, direct connections will not be possible.
## coder server
Workspaces connect to the coder server via the server's external address, set
via [`ACCESS_URL`](../../admin/setup/index.md#access-url). There must not be a
NAT between workspaces and coder server.
Users connect to the coder server's dashboard and API through its `ACCESS_URL`
as well. There must not be a NAT between users and the coder server.
Template admins can overwrite the site-wide access URL at the template level by
leveraging the `url` argument when
[defining the Coder provider](https://registry.terraform.io/providers/coder/coder/latest/docs#url):
```terraform
provider "coder" {
url = "https://coder.namespace.svc.cluster.local"
}
```
This is useful when debugging connectivity issues between the workspace agent
and the Coder server.
## Web Apps
The coder servers relays dashboard-initiated connections between the user and
the workspace. Web terminal <-> workspace connections are an exception and may
be direct.
In general, [port forwarded](./port-forwarding.md) web apps are faster than
dashboard-accessed web apps.
## 🌎 Geo-distribution
### Direct connections
Direct connections are a straight line between the user and workspace, so there
is no special geo-distribution configuration. To speed up direct connections,
move the user and workspace closer together.
Establishing a direct connection can be an involved process because both the
client and workspace agent will likely be behind at least one level of NAT,
meaning that we need to use STUN to learn the IP address and port under which
the client and agent can both contact each other. See [STUN and NAT](./stun.md)
for more information on how this process works.
If a direct connection is not available (e.g. client or server is behind NAT),
Coder will use a relayed connection. By default,
[Coder uses Google's public STUN server](../../reference/cli/server.md#--derp-server-stun-addresses),
but this can be disabled or changed for
[offline deployments](../../install/offline.md).
### Relayed connections
By default, your Coder server also runs a built-in DERP relay which can be used
for both public and [offline deployments](../../install/offline.md).
However, Tailscale has graciously allowed us to use
[their global DERP relays](https://tailscale.com/kb/1118/custom-derp-servers/#what-are-derp-servers).
You can launch `coder server` with Tailscale's DERPs like so:
```bash
$ coder server --derp-config-url https://controlplane.tailscale.com/derpmap/default
```
#### Custom Relays
If you want lower latency than what Tailscale offers or want additional DERP
relays for offline deployments, you may run custom DERP servers. Refer to
[Tailscale's documentation](https://tailscale.com/kb/1118/custom-derp-servers/#why-run-your-own-derp-server)
to learn how to set them up.
After you have custom DERP servers, you can launch Coder with them like so:
```json
# derpmap.json
{
"Regions": {
"1": {
"RegionID": 1,
"RegionCode": "myderp",
"RegionName": "My DERP",
"Nodes": [
{
"Name": "1",
"RegionID": 1,
"HostName": "your-hostname.com"
}
]
}
}
}
```
```bash
$ coder server --derp-config-path derpmap.json
```
### Dashboard connections
The dashboard (and web apps opened through the dashboard) are served from the
coder server, so they can only be geo-distributed with High Availability mode in
our Enterprise Edition. [Reach out to Sales](https://coder.com/contact) to learn
more.
## Browser-only connections (enterprise) (premium)
Some Coder deployments require that all access is through the browser to comply
with security policies. In these cases, pass the `--browser-only` flag to
`coder server` or set `CODER_BROWSER_ONLY=true`.
With browser-only connections, developers can only connect to their workspaces
via the web terminal and
[web IDEs](../../user-guides/workspace-access/web-ides.md).
### Workspace Proxies (enterprise) (premium)
Workspace proxies are a Coder Enterprise feature that allows you to provide
low-latency browser experiences for geo-distributed teams.
To learn more, see [Workspace Proxies](./workspace-proxies.md).
## Up next
- Learn about [Port Forwarding](./port-forwarding.md)
- Troubleshoot [Networking Issues](./troubleshooting.md)
+286
View File
@@ -0,0 +1,286 @@
# Port Forwarding
Port forwarding lets developers securely access processes on their Coder
workspace from a local machine. A common use case is testing web applications in
a browser.
There are three ways to forward ports in Coder:
- The `coder port-forward` command
- Dashboard
- SSH
The `coder port-forward` command is generally more performant than:
1. The Dashboard which proxies traffic through the Coder control plane versus
peer-to-peer which is possible with the Coder CLI
1. `sshd` which does double encryption of traffic with both Wireguard and SSH
## The `coder port-forward` command
This command can be used to forward TCP or UDP ports from the remote workspace
so they can be accessed locally. Both the TCP and UDP command line flags
(`--tcp` and `--udp`) can be given once or multiple times.
The supported syntax variations for the `--tcp` and `--udp` flag are:
- Single port with optional remote port: `local_port[:remote_port]`
- Comma separation `local_port1,local_port2`
- Port ranges `start_port-end_port`
- Any combination of the above
### Examples
Forward the remote TCP port `8080` to local port `8000`:
```console
coder port-forward myworkspace --tcp 8000:8080
```
Forward the remote TCP port `3000` and all ports from `9990` to `9999` to their
respective local ports.
```console
coder port-forward myworkspace --tcp 3000,9990-9999
```
For more examples, see `coder port-forward --help`.
## Dashboard
> To enable port forwarding via the dashboard, Coder must be configured with a
> [wildcard access URL](../../admin/setup/index.md#wildcard-access-url). If an
> access URL is not specified, Coder will create
> [a publicly accessible URL](../../admin/setup/index.md#tunnel) to reverse
> proxy the deployment, and port forwarding will work.
>
> There is a
> [DNS limitation](https://datatracker.ietf.org/doc/html/rfc1035#section-2.3.1)
> where each segment of hostnames must not exceed 63 characters. If your app
> name, agent name, workspace name and username exceed 63 characters in the
> hostname, port forwarding via the dashboard will not work.
### From an coder_app resource
One way to port forward is to configure a `coder_app` resource in the
workspace's template. This approach shows a visual application icon in the
dashboard. See the following `coder_app` example for a Node React app and note
the `subdomain` and `share` settings:
```tf
# node app
resource "coder_app" "node-react-app" {
agent_id = coder_agent.dev.id
slug = "node-react-app"
icon = "https://upload.wikimedia.org/wikipedia/commons/a/a7/React-icon.svg"
url = "http://localhost:3000"
subdomain = true
share = "authenticated"
healthcheck {
url = "http://localhost:3000/healthz"
interval = 10
threshold = 30
}
}
```
Valid `share` values include `owner` - private to the user, `authenticated` -
accessible by any user authenticated to the Coder deployment, and `public` -
accessible by users outside of the Coder deployment.
![Port forwarding from an app in the UI](../../images/networking/portforwarddashboard.png)
## Accessing workspace ports
Another way to port forward in the dashboard is to use the "Open Ports" button
to specify an arbitrary port. Coder will also detect if apps inside the
workspace are listening on ports, and list them below the port input (this is
only supported on Windows and Linux workspace agents).
![Port forwarding in the UI](../../images/networking/listeningports.png)
### Sharing ports
We allow developers to share ports as URLs, either with other authenticated
coder users or publicly. Using the open ports interface, developers can assign a
sharing levels that match our `coder_app`s share option in
[Coder terraform provider](https://registry.terraform.io/providers/coder/coder/latest/docs/resources/app#share).
- `owner` (Default): The implicit sharing level for all listening ports, only
visible to the workspace owner
- `authenticated`: Accessible by other authenticated Coder users on the same
deployment.
- `public`: Accessible by any user with the associated URL.
Once a port is shared at either `authenticated` or `public` levels, it will stay
pinned in the open ports UI for better accessibility regardless of whether or
not it is still accessible.
![Annotated port controls in the UI](../../images/networking/annotatedports.png)
The sharing level is limited by the maximum level enforced in the template
settings in enterprise deployments, and not restricted in OSS deployments.
This can also be used to change the sharing level of `coder_app`s by entering
their port number in the sharable ports UI. The `share` attribute on `coder_app`
resource uses a different method of authentication and **is not impacted by the
template's maximum sharing level**, nor the level of a shared port that points
to the app.
### Configure maximum port sharing level (enterprise) (premium)
Enterprise-licensed template admins can control the maximum port sharing level
for workspaces under a given template in the template settings. By default, the
maximum sharing level is set to `Owner`, meaning port sharing is disabled for
end-users. OSS deployments allow all workspaces to share ports at both the
`authenticated` and `public` levels.
![Max port sharing level in the UI](../../images/networking/portsharingmax.png)
### Configuring port protocol
Both listening and shared ports can be configured to use either `HTTP` or
`HTTPS` to connect to the port. For listening ports the protocol selector
applies to any port you input or select from the menu. Shared ports have
protocol configuration for each shared port individually.
You can access any port on the workspace and can configure the port protocol
manually by appending a `s` to the port in the URL.
```
# Uses HTTP
https://33295--agent--workspace--user--apps.example.com/
# Uses HTTPS
https://33295s--agent--workspace--user--apps.example.com/
```
### Cross-origin resource sharing (CORS)
When forwarding via the dashboard, Coder automatically sets headers that allow
requests between separately forwarded applications belonging to the same user.
When forwarding through other methods the application itself will need to set
its own CORS headers if they are being forwarded through different origins since
Coder does not intercept these cases. See below for the required headers.
#### Authentication
Since ports forwarded through the dashboard are private, cross-origin requests
must include credentials (set `credentials: "include"` if using `fetch`) or the
requests cannot be authenticated and you will see an error resembling the
following:
> Access to fetch at
> 'https://coder.example.com/api/v2/applications/auth-redirect' from origin
> 'https://8000--dev--user--apps.coder.example.com' has been blocked by CORS
> policy: No 'Access-Control-Allow-Origin' header is present on the requested
> resource. If an opaque response serves your needs, set the request's mode to
> 'no-cors' to fetch the resource with CORS disabled.
#### Headers
Below is a list of the cross-origin headers Coder sets with example values:
```
access-control-allow-credentials: true
access-control-allow-methods: PUT
access-control-allow-headers: X-Custom-Header
access-control-allow-origin: https://8000--dev--user--apps.coder.example.com
vary: Origin
vary: Access-Control-Request-Method
vary: Access-Control-Request-Headers
```
The allowed origin will be set to the origin provided by the browser if the
users are identical. Credentials are allowed and the allowed methods and headers
will echo whatever the request sends.
#### Configuration
These cross-origin headers are not configurable by administrative settings.
If applications set any of the above headers they will be stripped from the
response except for `Vary` headers that are set to a value other than the ones
listed above.
In other words, CORS behavior through the dashboard is not currently
configurable by either admins or users.
#### Allowed by default
<table class="tg">
<thead>
<tr>
<th class="tg-0pky" rowspan="2"></th>
<th class="tg-0pky" rowspan="3"></th>
<th class="tg-0pky">From</th>
<th class="tg-0pky" colspan="3">Alice</th>
<th class="tg-0pky">Bob</th>
</tr>
<tr>
<th class="tg-0pky" rowspan="2"></th>
<th class="tg-0pky">Workspace 1</th>
<th class="tg-0pky" colspan="2">Workspace 2</th>
<th class="tg-0pky">Workspace 3</th>
</tr>
<tr>
<th class="tg-0pky">To</th>
<th class="tg-0pky">App A</th>
<th class="tg-0pky">App B</th>
<th class="tg-0pky">App C</th>
<th class="tg-0pky">App D</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tg-0pky" rowspan="3">Alice</td>
<td class="tg-0pky" rowspan="2">Workspace 1</td>
<td class="tg-0pky">App A</td>
<td class="tg-0pky">✅</td>
<td class="tg-0pky">✅<span style="font-weight:400;font-style:normal">*</span></td>
<td class="tg-0pky">✅<span style="font-weight:400;font-style:normal">*</span></td>
<td class="tg-0pky">❌</td>
</tr>
<tr>
<td class="tg-0pky">App B</td>
<td class="tg-0pky">✅*</td>
<td class="tg-0pky">✅</td>
<td class="tg-0pky">✅<span style="font-weight:400;font-style:normal">*</span></td>
<td class="tg-0pky">❌</td>
</tr>
<tr>
<td class="tg-0pky">Workspace 2</td>
<td class="tg-0pky">App C</td>
<td class="tg-0pky">✅<span style="font-weight:400;font-style:normal">*</span></td>
<td class="tg-0pky">✅<span style="font-weight:400;font-style:normal">*</span></td>
<td class="tg-0pky">✅</td>
<td class="tg-0pky">❌</td>
</tr>
<tr>
<td class="tg-0pky">Bob</td>
<td class="tg-0pky">Workspace 3</td>
<td class="tg-0pky">App D</td>
<td class="tg-0pky">❌</td>
<td class="tg-0pky">❌</td>
<td class="tg-0pky">❌</td>
<td class="tg-0pky">✅</td>
</tr>
</tbody>
</table>
> '\*' means `credentials: "include"` is required
## SSH
First,
[configure SSH](../../user-guides/workspace-access/index.md#configure-ssh) on
your local machine. Then, use `ssh` to forward like so:
```console
ssh -L 8080:localhost:8000 coder.myworkspace
```
You can read more on SSH port forwarding
[here](https://www.ssh.com/academy/ssh/tunneling/example).
+174
View File
@@ -0,0 +1,174 @@
# STUN and NAT
> [Session Traversal Utilities for NAT (STUN)](https://www.rfc-editor.org/rfc/rfc8489.html)
> is a protocol used to assist applications in establishing peer-to-peer
> communications across Network Address Translations (NATs) or firewalls.
>
> [Network Address Translation (NAT)](https://en.wikipedia.org/wiki/Network_address_translation)
> is commonly used in private networks to allow multiple devices to share a
> single public IP address. The vast majority of home and corporate internet
> connections use at least one level of NAT.
## Overview
In order for one application to connect to another across a network, the
connecting application needs to know the IP address and port under which the
target application is reachable. If both applications reside on the same
network, then they can most likely connect directly to each other. In the
context of a Coder workspace agent and client, this is generally not the case,
as both agent and client will most likely be running in different _private_
networks (e.g. `192.168.1.0/24`). In this case, at least one of the two will
need to know an IP address and port under which they can reach their
counterpart.
This problem is often referred to as NAT traversal, and Coder uses a standard
protocol named STUN to address this.
Inside of that network, packets from the agent or client will show up as having
source address `192.168.1.X:12345`. However, outside of this private network,
the source address will show up differently (for example, `12.3.4.56:54321`). In
order for the Coder client and agent to establish a direct connection with each
other, one of them needs to know the `ip:port` pair under which their
counterpart can be reached. Once communication succeeds in one direction, we can
inspect the source address of the received packet to determine the return
address.
At a high level, STUN works like this:
> The below glosses over a lot of the complexity of traversing NATs. For a more
> in-depth technical explanation, see
> [How NAT traversal works (tailscale.com)](https://tailscale.com/blog/how-nat-traversal-works).
- **Discovery:** Both the client and agent will send UDP traffic to one or more
configured STUN servers. These STUN servers are generally located on the
public internet, and respond with the public IP address and port from which
the request came.
- **Coordination:** The client and agent then exchange this information through
the Coder server. They will then construct packets that should be able to
successfully traverse their counterpart's NATs successfully.
- **NAT Traversal:** The client and agent then send these crafted packets to
their counterpart's public addresses. If all goes well, the NATs on the other
end should route these packets to the correct internal address.
- **Connection:** Once the packets reach the other side, they send a response
back to the source `ip:port` from the packet. Again, the NATs should recognize
these responses as belonging to an ongoing communication, and forward them
accordingly.
At this point, both the client and agent should be able to send traffic directly
to each other.
## Examples
### 1. Direct connections without NAT or STUN
In this example, both the client and agent are located on the network
`192.168.21.0/24`. Assuming no firewalls are blocking packets in either
direction, both client and agent are able to communicate directly with each
other's locally assigned IP address.
![Diagram of a workspace agent and client in the same network](../../images/networking/stun1.png)
### 2. Direct connections with one layer of NAT
In this example, client and agent are located on different networks and connect
to each other over the public Internet. Both client and agent connect to a
configured STUN server located on the public Internet to determine the public IP
address and port on which they can be reached.
![Diagram of a workspace agent and client in separate networks](../../images/networking/stun2.1.png)
They then exchange this information through Coder server, and can then
communicate directly with each other through their respective NATs.
![Diagram of a workspace agent and client in separate networks](../../images/networking/stun2.2.png)
### 3. Direct connections with VPN and NAT hairpinning
In this example, the client workstation must use a VPN to connect to the
corporate network. All traffic from the client will enter through the VPN entry
node and exit at the VPN exit node inside the corporate network. Traffic from
the client inside the corporate network will appear to be coming from the IP
address of the VPN exit node `172.16.1.2`. Traffic from the client to the public
internet will appear to have the public IP address of the corporate router
`12.34.56.7`.
The workspace agent is running on a Kubernetes cluster inside the corporate
network, which is behind its own layer of NAT. To anyone inside the corporate
network but outside the cluster network, its traffic will appear to be coming
from `172.16.1.254`. However, traffic from the agent to services on the public
Internet will also see traffic originating from the public IP address assigned
to the corporate router. Additionally, the corporate router will most likely
have a firewall configured to block traffic from the internet to the corporate
network.
If the client and agent both use the public STUN server, the addresses
discovered by STUN will both be the public IP address of the corporate router.
To correctly route the traffic backwards, the corporate router must correctly
route both:
- Traffic sent from the client to the external IP of the corporate router back
to the cluster router, and
- Traffic sent from the agent to the external IP of the corporate router to the
VPN exit node.
This behaviour is known as "hairpinning", and may not be supported in all
network configurations.
If hairpinning is not supported, deploying an internal STUN server can aid
establishing direct connections between client and agent. When the agent and
client query this internal STUN server, they will be able to determine the
addresses on the corporate network from which their traffic appears to
originate. Using these internal addresses is much more likely to result in a
successful direct connection.
![Diagram of a workspace agent and client over VPN](../../images/networking/stun3.png)
## Hard NAT
Some NATs are known to use a different port when forwarding requests to the STUN
server and when forwarding probe packets to peers. In that case, the address a
peer discovers over the STUN protocol will have the correct IP address, but the
wrong port. Tailscale refers to this as "hard" NAT in
[How NAT traversal works (tailscale.com)](https://tailscale.com/blog/how-nat-traversal-works).
If both peers are behind a "hard" NAT, direct connections may take longer to
establish or will not be established at all. If one peer is behind a "hard" NAT
and the other is running a firewall (including Windows Defender Firewall), the
firewall may block direct connections.
In both cases, peers fallback to DERP connections if they cannot establish a
direct connection.
If your workspaces are behind a "hard" NAT, you can:
1. Ensure clients are not also behind a "hard" NAT. You may have limited ability
to control this if end users connect from their homes.
2. Ensure firewalls on client devices (e.g. Windows Defender Firewall) have an
inbound policy allowing all UDP ports either to the `coder` or `coder.exe`
CLI binary, or from the IP addresses of your workspace NATs.
3. Reconfigure your workspace network's NAT connection to the public internet to
be an "easy" NAT. See below for specific examples.
### AWS NAT Gateway
The
[AWS NAT Gateway](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat-gateway.html)
is a known "hard" NAT. You can use a
[NAT Instance](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_NAT_Instance.html)
instead of a NAT Gateway, and configure it to use the same port assignment for
all UDP traffic from a particular source IP:port combination (Tailscale calls
this "easy" NAT). Linux `MASQUERADE` rules work well for this.
### AWS Elastic Kubernetes Service (EKS)
The default configuration of AWS Elastic Kubernetes Service (EKS) includes the
[Amazon VPC CNI Driver](https://github.com/aws/amazon-vpc-cni-k8s), which by
default randomizes the public port for different outgoing UDP connections. This
makes it act as a "hard" NAT, even if the EKS nodes are on a public subnet (and
thus do not need to use the AWS NAT Gateway to reach the Internet).
This behavior can be disabled by setting the environment variable
`AWS_VPC_K8S_CNI_RANDOMIZESNAT=none` in the `aws-node` DaemonSet. Note, however,
if your nodes are on a private subnet, they will still need NAT to reach the
public Internet, meaning that issues with the
[AWS NAT Gateway](#aws-nat-gateway) might affect you.
+124
View File
@@ -0,0 +1,124 @@
# Troubleshooting
`coder ping <workspace>` will ping the workspace agent and print diagnostics on
the state of the connection. These diagnostics are created by inspecting both
the client and agent network configurations, and provide insights into why a
direct connection may be impeded, or why the quality of one might be degraded.
The `-v/--verbose` flag can be appended to the command to print client debug
logs.
```console
$ coder ping dev
pong from workspace proxied via DERP(Council Bluffs, Iowa) in 42ms
pong from workspace proxied via DERP(Council Bluffs, Iowa) in 41ms
pong from workspace proxied via DERP(Council Bluffs, Iowa) in 39ms
✔ preferred DERP region: 999 (Council Bluffs, Iowa)
✔ sent local data to Coder networking coordinator
✔ received remote agent data from Coder networking coordinator
preferred DERP region: 999 (Council Bluffs, Iowa)
endpoints: x.x.x.x:46433, x.x.x.x:46433, x.x.x.x:46433
✔ Wireguard handshake 11s ago
❗ You are connected via a DERP relay, not directly (p2p)
Possible client-side issues with direct connection:
- Network interface utun0 has MTU 1280, (less than 1378), which may degrade the quality of direct connections
Possible agent-side issues with direct connection:
- Agent is potentially behind a hard NAT, as multiple endpoints were retrieved from different STUN servers
- Agent IP address is within an AWS range (AWS uses hard NAT)
```
## Common Problems with Direct Connections
### Disabled Deployment-wide
Direct connections can be disabled at the deployment level by setting the
`CODER_BLOCK_DIRECT` environment variable or the `--block-direct-connections`
flag on the server. When set, this will be reflected in the output of
`coder ping`.
### UDP Blocked
Some corporate firewalls block UDP traffic. Direct connections require UDP
traffic to be allowed between the client and agent, as well as between the
client/agent and STUN servers in most cases. `coder ping` will indicate if
either the Coder agent or client had issues sending or receiving UDP packets to
STUN servers.
If this is the case, you may need to add exceptions to the firewall to allow UDP
for Coder workspaces, clients, and STUN servers.
### Endpoint-Dependent NAT (Hard NAT)
Hard NATs prevent public endpoints gathered from STUN servers from being used by
the peer to establish a direct connection. Typically, if only one side of the
connection is behind a hard NAT, direct connections can still be established
easily. However, if both sides are behind hard NATs, direct connections may take
longer to establish or may not be possible at all.
`coder ping` will indicate if it's possible the client or agent is behind a hard
NAT.
Learn more about [STUN and NAT](./stun.md).
### No STUN Servers
If there are no STUN servers available within a deployment's DERP MAP, direct
connections may not be possible. Notable exceptions are if the client and agent
are on the same network, or if either is able to use UPnP instead of STUN to
resolve the public IP of the other. `coder ping` will indicate if no STUN
servers were found.
### Endpoint Firewalls
Direct connections may also be impeded if one side is behind a hard NAT and the
other is running a firewall that blocks ingress traffic from unknown 5-tuples
(Protocol, Source IP, Source Port, Destination IP, Destination Port).
If this is suspected, you may need to add an exception for Coder to the
firewall, or reconfigure the hard NAT.
### VPNs
If a VPN is the default route for all IP traffic, it may interfere with the
ability for clients and agents to form direct connections. This happens if the
NAT does not permit traffic to be
['hairpinned'](./stun.md#3-direct-connections-with-vpn-and-nat-hairpinning) from
the public IP address of the NAT (determined via STUN) to the internal IP
address of the agent.
If this is the case, you may need to add exceptions to the VPN for Coder, modify
the NAT configuration, or deploy an internal STUN server.
### Low MTU
If a network interface on the side of either the client or agent has an MTU
smaller than 1378, any direct connections form may have degraded quality or
performance, as IP packets are fragmented. `coder ping` will indicate if this is
the case by inspecting network interfaces on both the client and the workspace
agent.
If another interface cannot be used, and the MTU cannot be changed, you may need
to disable direct connections, and relay all traffic via DERP instead, which
will not be affected by the low MTU.
## Throughput
The `coder speedtest <workspace>` command measures the throughput between the
client and the workspace agent.
```console
$ coder speedtest workspace
29ms via coder
Starting a 5s download test...
INTERVAL TRANSFER BANDWIDTH
0.00-1.00 sec 630.7840 MBits 630.7404 Mbits/sec
1.00-2.00 sec 913.9200 MBits 913.8106 Mbits/sec
2.00-3.00 sec 943.1040 MBits 943.0399 Mbits/sec
3.00-4.00 sec 933.3760 MBits 933.2143 Mbits/sec
4.00-5.00 sec 848.8960 MBits 848.7019 Mbits/sec
5.00-5.02 sec 13.5680 MBits 828.8189 Mbits/sec
----------------------------------------------------
0.00-5.02 sec 4283.6480 MBits 853.8217 Mbits/sec
```
+220
View File
@@ -0,0 +1,220 @@
# Workspace Proxies
Workspace proxies provide low-latency experiences for geo-distributed teams.
Coder's networking does a best effort to make direct connections to a workspace.
In situations where this is not possible, such as connections via the web
terminal and [web IDEs](../../user-guides/workspace-access/index.md#web-ides),
workspace proxies are able to reduce the amount of distance the network traffic
needs to travel.
A workspace proxy is a relay connection a developer can choose to use when
connecting with their workspace over SSH, a workspace app, port forwarding, etc.
Dashboard connections and API calls (e.g. the workspaces list) are not served
over workspace proxies.
![ProxyDiagram](../../images/admin/networking/workspace-proxies/proxydiagram.png)
# Deploy a workspace proxy
Each workspace proxy should be a unique instance. At no point should 2 workspace
proxy instances share the same authentication token. They only require port 443
to be open and are expected to have network connectivity to the coderd
dashboard. Workspace proxies **do not** make any database connections.
Workspace proxies can be used in the browser by navigating to the user
`Account -> Workspace Proxy`
## Requirements
- The [Coder CLI](../../reference/cli/index.md) must be installed and
authenticated as a user with the Owner role.
## Step 1: Create the proxy
Create the workspace proxy and make sure to save the returned authentication
token for said proxy. This is the token the workspace proxy will use to
authenticate back to primary coderd.
```bash
$ coder wsproxy create --name=newyork --display-name="USA East" --icon="/emojis/2194.png"
Workspace Proxy "newyork" created successfully. Save this token, it will not be shown again.
Token: 2fb6500b-bb47-4783-a0db-dedde895b865:05271b4ef9432bac14c02b3c56b5a2d7f05453718a1f85ba7e772c0a096c7175
```
To verify it was created.
```bash
$ coder wsproxy ls
NAME URL STATUS STATUS
newyork unregistered
```
## Step 2: Deploy the proxy
Deploying the workspace proxy will also register the proxy with coderd and make
the workspace proxy usable. If the proxy deployment is successful,
`coder wsproxy ls` will show an `ok` status code:
```
$ coder wsproxy ls
NAME URL STATUS STATUS
brazil-saopaulo https://brazil.example.com ok
europe-frankfurt https://europe.example.com ok
sydney https://sydney.example.com ok
```
Other Status codes:
- `unregistered` : The workspace proxy was created, and not yet deployed
- `unreachable` : The workspace proxy was registered, but is not responding.
Likely the proxy went offline.
- `unhealthy` : The workspace proxy is reachable, but has some issue that is
preventing the proxy from being used. `coder wsproxy ls` should show the error
message.
- `ok` : The workspace proxy is healthy and working properly!
### Configuration
Workspace proxy configuration overlaps with a subset of the coderd
configuration. To see the full list of configuration options:
`coder wsproxy server --help`
```bash
# Proxy specific configuration. These are REQUIRED
# Example: https://coderd.example.com
CODER_PRIMARY_ACCESS_URL="https://<url_of_coderd_dashboard>"
CODER_PROXY_SESSION_TOKEN="<session_token_from_proxy_create>"
# Runtime variables for "coder start".
CODER_HTTP_ADDRESS=0.0.0.0:80
CODER_TLS_ADDRESS=0.0.0.0:443
# Example: https://east.coderd.example.com
CODER_ACCESS_URL="https://<access_url_of_proxy>"
# Example: *.east.coderd.example.com
CODER_WILDCARD_ACCESS_URL="*.<app_hostname_of_proxy>"
CODER_TLS_ENABLE=true
CODER_TLS_CLIENT_AUTH=none
CODER_TLS_CERT_FILE="<cert_file_location>"
CODER_TLS_KEY_FILE="<key_file_location>"
# Additional configuration options are available.
```
### Running on Kubernetes
Make a `values-wsproxy.yaml` with the workspace proxy configuration:
> Notice the `workspaceProxy` configuration which is `false` by default in the
> coder Helm chart.
```yaml
coder:
env:
- name: CODER_PRIMARY_ACCESS_URL
value: "https://<url_of_coderd_dashboard>"
- name: CODER_PROXY_SESSION_TOKEN
value: "<session_token_from_proxy_create>"
# Example: https://east.coderd.example.com
- name: CODER_ACCESS_URL
value: "https://<access_url_of_proxy>"
# Example: *.east.coderd.example.com
- name: CODER_WILDCARD_ACCESS_URL
value: "*.<app_hostname_of_proxy>"
tls:
secretNames:
- kubernetes-wsproxy-secret
# enable workspace proxy
workspaceProxy: true
```
Using Helm, install the workspace proxy chart
```bash
helm install coder coder-v2/coder --namespace <your workspace proxy namespace> -f ./values-wsproxy.yaml
```
Test that the workspace proxy is reachable with `curl -vvv`. If for some reason,
the Coder dashboard still shows the workspace proxy is `UNHEALTHY`, scale down
and up the deployment's replicas.
### Running on a VM
```bash
# Set configuration options via environment variables, a config file, or cmd flags
coder wsproxy server
```
### Running as a system service
If you've installed Coder via a [system package](../../install/index.md), you
can configure the workspace proxy by settings in
`/etc/coder.d/coder-workspace-proxy.env`
To run workspace proxy as a system service on the host:
```bash
# Use systemd to start workspace proxy now and on reboot
sudo systemctl enable --now coder-workspace-proxy
# View the logs to ensure a successful start
journalctl -u coder-workspace-proxy.service -b
```
To restart workspace proxy after applying system changes:
```shell
sudo systemctl restart coder-workspace-proxy
```
### Running in Docker
Modify the default entrypoint to run a workspace proxy server instead of a
regular Coder server.
#### Docker Compose
Change the provided
[`docker-compose.yml`](https://github.com/coder/coder/blob/main/docker-compose.yaml)
file to include a custom entrypoint:
```diff
image: ghcr.io/coder/coder:${CODER_VERSION:-latest}
+ entrypoint: /opt/coder wsproxy server
```
#### Docker run
```bash
docker run --rm -it --entrypoint /opt/coder ghcr.io/coder/coder:latest wsproxy server
```
#### Custom Dockerfile
```Dockerfile
FROM ghcr.io/coder/coder:latest
ENTRYPOINT ["/opt/coder", "wsproxy", "server"]
```
### Selecting a proxy
Users can select a workspace proxy at the top-right of the browser-based Coder
dashboard. Workspace proxy preferences are cached by the web browser. If a proxy
goes offline, the session will fall back to the primary proxy. This could take
up to 60 seconds.
![Workspace proxy picker](../../images/admin/networking/workspace-proxies/ws-proxy-picker.png)
## Observability
Coder workspace proxy exports metrics via the HTTP endpoint, which can be
enabled using either the environment variable `CODER_PROMETHEUS_ENABLE` or the
flag `--prometheus-enable`.
The Prometheus endpoint address is `http://localhost:2112/` by default. You can
use either the environment variable `CODER_PROMETHEUS_ADDRESS` or the flag
`--prometheus-address <network-interface>:<port>` to select a different listen
address.