shishantbiswas/coder
1
0
Fork 0
mirror of https://github.com/coder/coder.git synced 2026-06-02 20:48:20 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: add workspace process logging doc (#9002)

Browse Source
This commit is contained in:
Dean Sheather
2023-08-18 06:02:01 -07:00
committed by GitHub
parent 531f740070
commit f4f3132120
2 changed files with 317 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/templates/docker-in-workspaces.md
Vendored
+2
View File
@@ -13,6 +13,8 @@ There are a few ways to run Docker within container-based Coder workspaces.
The [Sysbox](https://github.com/nestybox/sysbox) container runtime allows unprivileged users to run system-level applications, such as Docker, securely from the workspace containers. Sysbox requires a [compatible Linux distribution](https://github.com/nestybox/sysbox/blob/master/docs/distro-compat.md) to implement these security features. Sysbox can also be used to run systemd inside Coder workspaces. See [Systemd in Docker](#systemd-in-docker).
The Sysbox container runtime is not compatible with our [workspace process logging](./process-logging.md) feature. Envbox is compatible with process logging, however.
### Use Sysbox in Docker-based templates
After [installing Sysbox](https://github.com/nestybox/sysbox#installation) on the Coder host, modify your template to use the sysbox-runc runtime:
docs/templates/process-logging.md
Vendored
+315
View File
@@ -0,0 +1,315 @@
# Workspace Process Logging
The workspace process logging feature allows you to log all system-level
processes executing in the workspace.
> **Note:** This feature is only available on Linux in Kubernetes. There are
> additional requirements outlined further in this document.
>
> This is an [Enterprise](https://coder.com/docs/v2/latest/enterprise) feature.
> To learn more about Coder Enterprise, please
> [contact sales](https://coder.com/contact).
Workspace process logging adds a sidecar container to workspace pods that will
log all processes started in the workspace container (e.g., commands executed in
the terminal or processes created in the background by other processes).
Processes launched inside containers or nested containers within the workspace
are also logged. You can view the output from the sidecar or send it to a
monitoring stack, such as CloudWatch, for further analysis or long-term storage.
Please note that these logs are not recorded or captured by the Coder
organization in any way, shape, or form.
## How this works
Coder uses [eBPF](https://ebpf.io/) (which we chose for its minimal performance
impact) to perform in-kernel logging and filtering of all exec system calls
originating from the workspace container.
The core of this feature is also open source and can be found in the
[exectrace](https://github.com/coder/exectrace) GitHub repo. The enterprise
component (in the `enterprise/` directory of the repo) is responsible for
starting the eBPF program with the correct filtering options for the specific
workspace.
## Requirements
The host machine must be running a Linux kernel >= 5.8 with the kernel config
`CONFIG_DEBUG_INFO_BTF=y` enabled.
To check your kernel version, run:
```shell
uname -r
```
To validate the required kernel config is enabled, run either of the following
commands on your nodes directly (_not_ from a workspace terminal):
```shell
cat /proc/config.gz | gunzip | grep CONFIG_DEBUG_INFO_BTF
```
```shell
cat "/boot/config-$(uname -r)" | grep CONFIG_DEBUG_INFO_BTF
```
If these requirements are not met, workspaces will fail to start for security
reasons.
Your template must be a Kubernetes template. Workspace process logging is not
compatible with the `sysbox-runc` runtime due to technical limitations, but it
is compatible with our `envbox` template family.
## Example templates
We provide working example templates for Kubernetes, and Kubernetes with
`envbox` (for [Docker support in workspaces](./docker-in-workspaces.md)). You
can view these templates in the
[exectrace repo](https://github.com/coder/exectrace/tree/main/enterprise/templates).
## Configuring custom templates to use workspace process logging
If you have an existing Kubernetes or Kubernetes with `envbox` template that you
would like to add workspace process logging to, follow these steps:
1. Ensure the image used in your template has `curl` installed.
1. Add the following section to your template's `main.tf` file:
<!--
If you are updating this section, please also update the example templates
in the exectrace repo.
-->
```hcl
locals {
# This is the init script for the main workspace container that runs before the
# agent starts to configure workspace process logging.
exectrace_init_script = <<EOT
set -eu
pidns_inum=$(readlink /proc/self/ns/pid | sed 's/[^0-9]//g')
if [ -z "$pidns_inum" ]; then
echo "Could not determine process ID namespace inum"
exit 1
fi
# Before we start the script, does curl exist?
if ! command -v curl >/dev/null 2>&1; then
echo "curl is required to download the Coder binary"
echo "Please install curl to your image and try again"
# 127 is command not found.
exit 127
fi
echo "Sending process ID namespace inum to exectrace sidecar"
rc=0
max_retry=5
counter=0
until [ $counter -ge $max_retry ]; do
set +e
curl \
--fail \
--silent \
--connect-timeout 5 \
-X POST \
-H "Content-Type: text/plain" \
--data "$pidns_inum" \
http://127.0.0.1:56123
rc=$?
set -e
if [ $rc -eq 0 ]; then
break
fi
counter=$((counter+1))
echo "Curl failed with exit code $${rc}, attempt $${counter}/$${max_retry}; Retrying in 3 seconds..."
sleep 3
done
if [ $rc -ne 0 ]; then
echo "Failed to send process ID namespace inum to exectrace sidecar"
exit $rc
fi
EOT
}
```
1. Update the `command` of your workspace container like the following:
<!--
If you are updating this section, please also update the example templates
in the exectrace repo.
-->
```hcl
resource "kubernetes_pod" "main" {
...
spec {
...
container {
...
// NOTE: this command is changed compared to the upstream kubernetes
// template
command = [
"sh",
"-c",
"${local.exectrace_init_script}\n\n${coder_agent.main.init_script}",
]
...
}
...
}
...
}
```
> **Note:** If you are using the `envbox` template, you will need to update
> the third argument to be
> `"${local.exectrace_init_script}\n\nexec /envbox docker"` instead.
1. Add the following container to your workspace pod spec.
<!--
If you are updating this section, please also update the example templates
in the exectrace repo.
-->
```hcl
resource "kubernetes_pod" "main" {
...
spec {
...
// NOTE: this container is added compared to the upstream kubernetes
// template
container {
name = "exectrace"
image = "ghcr.io/coder/exectrace:latest"
image_pull_policy = "Always"
command = [
"/opt/exectrace",
"--init-address", "127.0.0.1:56123",
"--label", "workspace_id=${data.coder_workspace.me.id}",
"--label", "workspace_name=${data.coder_workspace.me.name}",
"--label", "user_id=${data.coder_workspace.me.owner_id}",
"--label", "username=${data.coder_workspace.me.owner}",
"--label", "user_email=${data.coder_workspace.me.owner_email}",
]
security_context {
// exectrace must be started as root so it can attach probes into the
// kernel to record process events with high throughput.
run_as_user = "0"
run_as_group = "0"
// exectrace requires a privileged container so it can control mounts
// and perform privileged syscalls against the host kernel to attach
// probes.
privileged = true
}
}
...
}
...
}
```
> **Note:** `exectrace` requires root privileges and a privileged container
> to attach probes to the kernel. This is a requirement of eBPF.
1. Add the following environment variable to your workspace pod:
<!--
If you are updating this section, please also update the example templates
in the exectrace repo.
-->
```hcl
resource "kubernetes_pod" "main" {
...
spec {
...
env {
name = "CODER_AGENT_SUBSYSTEM"
value = "exectrace"
}
...
}
...
}
```
Once you have made these changes, you can push a new version of your template
and workspace process logging will be enabled for all workspaces once they are
restarted.
## Viewing workspace process logs
To view the process logs for a specific workspace you can use `kubectl` to print
the logs:
```bash
kubectl logs pod-name --container exectrace
```
The raw logs will look something like this:
```json
{
"ts": "2022-02-28T20:29:38.038452202Z",
"level": "INFO",
"msg": "exec",
"fields": {
"labels": {
"user_email": "jessie@coder.com",
"user_id": "5e876e9a-121663f01ebd1522060d5270",
"username": "jessie",
"workspace_id": "621d2e52-a6987ef6c56210058ee2593c",
"workspace_name": "main"
},
"cmdline": "uname -a",
"event": {
"filename": "/usr/bin/uname",
"argv": ["uname", "-a"],
"truncated": false,
"pid": 920684,
"uid": 101000,
"gid": 101000,
"comm": "bash"
}
}
}
```
### View logs in AWS EKS
If you're using AWS' Elastic Kubernetes Service, you can
[configure your cluster](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-EKS-logs.html)
to send logs to CloudWatch. This allows you to view the logs for a specific user
or workspace.
To view your logs, go to the CloudWatch dashboard (which is available on the
**Log Insights** tab) and run a query similar to the following:
```text
fields @timestamp, log_processed.fields.cmdline
| sort @timestamp asc
| filter kubernetes.container_name="exectrace"
| filter log_processed.fields.labels.username="zac"
| filter log_processed.fields.labels.workspace_name="code"
```
## Usage considerations
- The sidecar attached to each workspace is a privileged container, so you may
need to review your organization's security policies before enabling this
feature. Enabling workspace process logging does _not_ grant extra privileges
to the workspace container itself, however.
- `exectrace` will log processes from nested Docker containers (including deeply
nested containers) correctly, but Coder does not distinguish between processes
started in the workspace and processes started in a child container in the
logs.
- With `envbox` workspaces, this feature will detect and log startup processes
begun in the outer container (including container initialization processes).
- Because this feature logs **all** processes in the workspace, high levels of
usage (e.g., during a `make` run) will result in an abundance of output in the
sidecar container. Depending on how your Kubernetes cluster is configured, you
may incur extra charges from your cloud provider to store the additional logs.