shishantbiswas/coder
1
0
Fork 0
mirror of https://github.com/coder/coder.git synced 2026-06-02 20:48:20 +00:00
Code Issues Packages Projects Releases Wiki Activity

chore(examples): update incus template (#24616)

Browse Source 
- Decoupled provisioner from Incus host by passing agent token/URL via
Incus Guest API
- Added a config watcher service to detect token updates and restart the
agent automatically.
- Updates for compatibility with Incus provider 1.x
This commit is contained in:
Cian Johnston
2026-04-22 20:06:21 +01:00
committed by GitHub
parent 1ace519c6e
commit 68508e1fd3
2 changed files with 232 additions and 158 deletions
Download Patch File Download Diff File Expand all files Collapse all files
examples/templates/incus/README.md
+43 -14
View File
@@ -1,22 +1,42 @@
---
display_name: Incus System Container with Docker
description: Develop in an Incus System Container with Docker using incus
description: Develop in an Incus System Container with Docker using Incus
icon: ../../../site/static/icon/lxc.svg
maintainer_github: coder
verified: true
tags: [local, incus, lxc, lxd]
tags: [incus, lxc, lxd]
---
# Incus System Container with Docker
Develop in an Incus System Container and run nested Docker containers using Incus on your local infrastructure.
Develop in an Incus System Container and run nested Docker containers using Incus.
## Architecture
This template uses the [Incus guest API](https://linuxcontainers.org/incus/docs/main/dev-incus/) (`/dev/incus/sock`) to deliver the Coder agent token and URL into the container without any host filesystem coupling. This means:
- **The provisioner does not need to run on the Incus host.** There are no bind mounts or local file writes. All configuration is passed via Incus `user.*` config keys and read from inside the container at runtime.
- **The agent binary is downloaded automatically.** The standard Coder init script fetches the correct binary from the Coder server on every boot, keeping it in sync with the server version.
- **The agent token is refreshed on every start.** Terraform updates the `user.coder_agent_token` config key each workspace start. A watcher service inside the container listens for config changes via the guest API events endpoint and restarts the agent when a new token arrives.
### Boot sequence
1. **First boot (cloud-init):** Creates the workspace user, writes the bootstrap scripts and systemd units, installs `curl` and `git`, and enables the services. Cloud-init only runs once.
2. **Every boot (systemd):**
- `coder-agent-config.service` (oneshot) reads `CODER_AGENT_TOKEN` and `CODER_AGENT_URL` from the Incus guest API and writes them to `/opt/coder/init.env`.
- `coder-agent.service` loads the env file and runs the Coder init script, which downloads the agent binary and starts it.
- `coder-agent-watcher.service` streams config change events from the guest API. If the Incus provider updates the token *after* the container has already booted (a known provider ordering issue), the watcher detects the change, re-fetches the config, and restarts the agent.
### Packages
Essential packages (`curl`, `git`) are installed via cloud-init on first boot, before the agent starts. Additional packages (e.g. `docker.io`) are installed via a non-blocking [`coder_script`](https://registry.terraform.io/providers/coder/coder/latest/docs/resources/script) that runs on each workspace start. It does not block login; users can connect to the workspace immediately while packages install in the background. On subsequent starts, it detects packages are already installed and skips the installation.
## Prerequisites
1. Install [Incus](https://linuxcontainers.org/incus/) on the same machine as Coder.
1. Install [Incus](https://linuxcontainers.org/incus/) on a machine reachable by the Coder provisioner.
2. Allow Coder to access the Incus socket.
- If you're running Coder as system service, run `sudo usermod -aG incus-admin coder` and restart the Coder service.
- If you're running Coder as a system service, run `sudo usermod -aG incus-admin coder` and restart the Coder service.
- If you're running Coder as a Docker Compose service, get the group ID of the `incus-admin` group by running `getent group incus-admin` and add the following to your `compose.yaml` file:
```yaml
@@ -28,24 +48,33 @@ Develop in an Incus System Container and run nested Docker containers using Incu
- 996 # Replace with the group ID of the `incus-admin` group
```
3. Create a storage pool named `coder` and `btrfs` as the driver by running `incus storage create coder btrfs`.
3. Create a storage pool named `coder` by running `incus storage create coder btrfs` (or use another [supported driver](https://linuxcontainers.org/incus/docs/main/reference/storage_drivers/)).
## Usage
> **Note:** this template requires using a container image with cloud-init installed such as `ubuntu/jammy/cloud/amd64`.
> **Note:** This template requires a container image with cloud-init installed, such as `images:debian/13/cloud` or `images:ubuntu/24.04/cloud`. Images are pulled automatically from the [Linux Containers image server](https://images.linuxcontainers.org/).
1. Run `coder templates init -id incus`
1. Select this template
1. Follow the on-screen instructions
1. Run `coder templates push --directory .` from this directory.
2. Create a workspace from the template in the Coder UI.
## Parameters
| Parameter | Description | Default |
|--------------------|--------------------------------------------------------------------------------------------|--------------------------|
| **Image** | Container image with cloud-init. Options: Debian 13, Debian 12, Ubuntu 24.04, Ubuntu 22.04 | `images:debian/13/cloud` |
| **CPU** | Number of CPUs (1-8) | `1` |
| **Memory** | Memory in GB (1-16) | `2` |
| **Storage pool** | Incus storage pool name | `coder` |
| **Git repository** | Clone a git repo inside the workspace | *(empty)* |
## Extending this template
See the [lxc/incus](https://registry.terraform.io/providers/lxc/incus/latest/docs) Terraform provider documentation to
add the following features to your Coder template:
See the [lxc/incus](https://registry.terraform.io/providers/lxc/incus/latest/docs) Terraform provider documentation to add the following features to your Coder template:
- HTTPS incus host
- Volume mounts
- Remote Incus hosts (HTTPS)
- Additional volume mounts
- Custom networks
- GPU passthrough
- More
We also welcome contributions!
examples/templates/incus/main.tf
+189 -144
View File
@@ -1,10 +1,12 @@
terraform {
required_providers {
coder = {
source = "coder/coder"
source = "coder/coder"
version = "~>2"
}
incus = {
source = "lxc/incus"
source = "lxc/incus"
version = "~>1.0"
}
}
}
@@ -19,10 +21,28 @@ data "coder_workspace_owner" "me" {}
data "coder_parameter" "image" {
name = "image"
display_name = "Image"
description = "The container image to use. Be sure to use a variant with cloud-init installed!"
default = "ubuntu/jammy/cloud/amd64"
description = "The container image to use. Must have cloud-init installed."
default = "images:debian/13/cloud"
icon = "/icon/image.svg"
mutable = true
mutable = false
option {
name = "Debian 13 (Trixie)"
value = "images:debian/13/cloud"
}
option {
name = "Debian 12 (Bookworm)"
value = "images:debian/12/cloud"
}
option {
name = "Ubuntu 24.04 (Noble)"
value = "images:ubuntu/24.04/cloud"
}
option {
name = "Ubuntu 22.04 (Jammy)"
value = "images:ubuntu/22.04/cloud"
}
}
data "coder_parameter" "cpu" {
@@ -56,17 +76,18 @@ data "coder_parameter" "memory" {
data "coder_parameter" "git_repo" {
type = "string"
name = "Git repository"
default = "https://github.com/coder/coder"
description = "Clone a git repo into [base directory]"
default = ""
description = "Clone a git repo inside the workspace"
mutable = true
}
data "coder_parameter" "repo_base_dir" {
type = "string"
name = "Repository Base Directory"
default = "~"
description = "The directory specified will be created (if missing) and the specified repo will be cloned into [base directory]/{repo}🪄."
mutable = true
data "coder_parameter" "pool" {
type = "string"
name = "pool"
display_name = "Storage pool"
default = "coder"
description = "Incus storage pool name"
mutable = false
}
resource "coder_agent" "main" {
@@ -75,7 +96,9 @@ resource "coder_agent" "main" {
os = "linux"
dir = "/home/${local.workspace_user}"
env = {
CODER_WORKSPACE_ID = data.coder_workspace.me.id
CODER_WORKSPACE_ID = data.coder_workspace.me.id
CODER_SESSION_TOKEN = data.coder_workspace_owner.me.session_token
CODER_URL = data.coder_workspace.me.access_url
}
metadata {
@@ -93,87 +116,74 @@ resource "coder_agent" "main" {
interval = 10
timeout = 1
}
metadata {
display_name = "Home Disk"
key = "3_home_disk"
script = "coder stat disk --path /home/${lower(data.coder_workspace_owner.me.name)}"
interval = 60
timeout = 1
}
}
# https://registry.coder.com/modules/coder/git-clone
module "git-clone" {
source = "registry.coder.com/coder/git-clone/coder"
# This ensures that the latest non-breaking version of the module gets downloaded, you can also pin the module version to prevent breaking changes in production.
version = "~> 1.0"
agent_id = local.agent_id
url = data.coder_parameter.git_repo.value
base_dir = local.repo_base_dir
# Note: execution order is currently not guaranteed so only
# include packages here that are not required for either the
# agent or modules.
resource "coder_script" "packages" {
count = data.coder_workspace.me.start_count
agent_id = coder_agent.main[0].id
display_name = "Install packages"
icon = "/icon/debian.svg"
run_on_start = true
script = <<-EOF
#!/bin/bash
set -e
PACKAGES=(docker.io)
MISSING=()
for pkg in "$${PACKAGES[@]}"; do
if ! dpkg -s "$pkg" &> /dev/null; then
MISSING+=("$pkg")
fi
done
if [ "$${#MISSING[@]}" -gt 0 ]; then
echo "Installing: $${MISSING[*]}"
sudo apt-get update
sudo apt-get install -y "$${MISSING[@]}"
echo "Packages installed successfully"
else
echo "All packages already installed"
fi
# Ensure the workspace user can access the Docker socket without
# needing the docker group (which would require a new login session).
if [ -S /var/run/docker.sock ]; then
sudo chown $(whoami) /var/run/docker.sock
fi
EOF
}
# https://registry.coder.com/modules/coder/code-server
module "code-server" {
source = "registry.coder.com/coder/code-server/coder"
# This ensures that the latest non-breaking version of the module gets downloaded, you can also pin the module version to prevent breaking changes in production.
version = "~> 1.0"
agent_id = local.agent_id
folder = local.repo_base_dir
}
# https://registry.coder.com/modules/coder/filebrowser
module "filebrowser" {
source = "registry.coder.com/coder/filebrowser/coder"
# This ensures that the latest non-breaking version of the module gets downloaded, you can also pin the module version to prevent breaking changes in production.
version = "~> 1.0"
agent_id = local.agent_id
}
# https://registry.coder.com/modules/coder/coder-login
module "coder-login" {
source = "registry.coder.com/coder/coder-login/coder"
# This ensures that the latest non-breaking version of the module gets downloaded, you can also pin the module version to prevent breaking changes in production.
version = "~> 1.0"
agent_id = local.agent_id
}
resource "incus_volume" "home" {
resource "incus_storage_volume" "home" {
name = "coder-${data.coder_workspace.me.id}-home"
pool = local.pool
}
resource "incus_volume" "docker" {
name = "coder-${data.coder_workspace.me.id}-docker"
pool = local.pool
}
resource "incus_cached_image" "image" {
source_remote = "images"
source_image = data.coder_parameter.image.value
}
resource "incus_instance_file" "agent_token" {
count = data.coder_workspace.me.start_count
instance = incus_instance.dev.name
content = <<EOF
CODER_AGENT_TOKEN=${local.agent_token}
EOF
create_directories = true
target_path = "/opt/coder/init.env"
}
resource "incus_instance" "dev" {
running = data.coder_workspace.me.start_count == 1
name = "coder-${lower(data.coder_workspace_owner.me.name)}-${lower(data.coder_workspace.me.name)}"
image = incus_cached_image.image.fingerprint
image = data.coder_parameter.image.value
config = {
"limits.cpu" = data.coder_parameter.cpu.value
"limits.memory" = "${data.coder_parameter.memory.value}GiB"
"security.nesting" = true
"security.syscalls.intercept.mknod" = true
"security.syscalls.intercept.setxattr" = true
"boot.autostart" = true
"cloud-init.user-data" = <<EOF
# Pass the agent token and URL via Incus user config keys.
# These are readable from inside the container via the guest API at
# /dev/incus/sock, which removes the need for bind-mounting files from
# the host. This decouples the provisioner from the Incus host. They
# no longer need to share a filesystem. The token is refreshed by
# Terraform on every workspace start; Incus updates the config on the
# instance even while it is stopped, so the new value is available
# immediately when the container boots.
"user.coder_agent_token" = local.agent_token
"user.coder_agent_url" = data.coder_workspace.me.access_url
"cloud-init.user-data" = <<EOF
#cloud-config
hostname: ${lower(data.coder_workspace.me.name)}
users:
@@ -181,93 +191,119 @@ users:
uid: 1000
gid: 1000
groups: sudo
packages:
- curl
shell: /bin/bash
sudo: ['ALL=(ALL) NOPASSWD:ALL']
write_files:
- path: /opt/coder/init
# Pre-start script that reads the agent token and URL from the Incus guest
# API (/dev/incus/sock) and writes them to an env file. This runs as a
# separate oneshot unit (coder-agent-config.service) before the agent starts,
# ensuring the env file is always fresh. This approach:
# - Eliminates host filesystem coupling (no bind mounts for token or binary)
# - Allows the provisioner to run on a different host than Incus
# - Delivers a fresh agent token on every start without cloud-init re-runs
- path: /opt/coder/fetch-config.sh
permissions: "0755"
content: |
#!/bin/bash
set -euo pipefail
INCUS_SOCK="/dev/incus/sock"
# Read agent config from Incus guest API.
CODER_AGENT_TOKEN=$(curl -sf --unix-socket "$INCUS_SOCK" http://localhost/1.0/config/user.coder_agent_token)
CODER_AGENT_URL=$(curl -sf --unix-socket "$INCUS_SOCK" http://localhost/1.0/config/user.coder_agent_url)
# Write env file for the systemd service.
printf 'CODER_AGENT_TOKEN=%s\nCODER_AGENT_URL=%s\n' "$CODER_AGENT_TOKEN" "$CODER_AGENT_URL" > /opt/coder/init.env
# The standard Coder agent init script, provided by coder_agent.init_script.
# This handles downloading the correct agent binary and running it.
- path: /opt/coder/coder-init.sh
permissions: "0755"
encoding: b64
content: ${base64encode(local.agent_init_script)}
- path: /etc/systemd/system/coder-agent-config.service
permissions: "0644"
content: |
[Unit]
Description=Fetch Coder Agent Config from Incus Guest API
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
ExecStart=/opt/coder/fetch-config.sh
# Watcher script that listens for config changes via the Incus guest API
# events endpoint. The Incus Terraform provider starts the instance before
# updating config keys, so on a stop->start cycle the agent initially boots
# with a stale token. This watcher detects when user.coder_agent_token is
# updated, re-fetches the config, and restarts the agent with the new token.
- path: /opt/coder/watch-config.sh
permissions: "0755"
content: |
#!/bin/bash
INCUS_SOCK="/dev/incus/sock"
curl -sfN --unix-socket "$INCUS_SOCK" http://localhost/1.0/events?type=config | \
while read -r event; do
key=$(echo "$event" | sed -n 's/.*"key":"\([^"]*\)".*/\1/p')
if [ "$key" = "user.coder_agent_token" ]; then
/opt/coder/fetch-config.sh
systemctl restart coder-agent.service
fi
done
- path: /etc/systemd/system/coder-agent-watcher.service
permissions: "0644"
content: |
[Unit]
Description=Watch for Coder Agent config changes via Incus Guest API
After=network-online.target
Wants=network-online.target
[Service]
ExecStart=/opt/coder/watch-config.sh
Restart=always
RestartSec=5
[Install]
WantedBy=multi-user.target
- path: /etc/systemd/system/coder-agent.service
permissions: "0644"
content: |
[Unit]
Description=Coder Agent
After=network-online.target
After=network-online.target coder-agent-config.service
Wants=network-online.target
Requires=coder-agent-config.service
[Service]
User=${local.workspace_user}
EnvironmentFile=/opt/coder/init.env
ExecStart=/opt/coder/init
ExecStart=/opt/coder/coder-init.sh
Restart=always
RestartSec=10
TimeoutStopSec=90
KillMode=process
OOMScoreAdjust=-900
SyslogIdentifier=coder-agent
[Install]
WantedBy=multi-user.target
- path: /etc/systemd/system/coder-agent-watcher.service
permissions: "0644"
content: |
[Unit]
Description=Coder Agent Watcher
After=network-online.target
[Service]
Type=oneshot
ExecStart=/usr/bin/systemctl restart coder-agent.service
[Install]
WantedBy=multi-user.target
- path: /etc/systemd/system/coder-agent-watcher.path
permissions: "0644"
content: |
[Path]
PathModified=/opt/coder/init.env
Unit=coder-agent-watcher.service
[Install]
WantedBy=multi-user.target
runcmd:
- chown -R ${local.workspace_user}:${local.workspace_user} /home/${local.workspace_user}
- |
#!/bin/bash
apt-get update && apt-get install -y curl docker.io
usermod -aG docker ${local.workspace_user}
newgrp docker
- systemctl enable coder-agent.service coder-agent-watcher.service coder-agent-watcher.path
- systemctl start coder-agent.service coder-agent-watcher.service coder-agent-watcher.path
# Install package dependencies before starting the agent.
- apt-get update && apt-get install -y curl git
- systemctl daemon-reload
- systemctl enable coder-agent.service coder-agent-watcher.service
- systemctl start coder-agent.service coder-agent-watcher.service
EOF
}
limits = {
cpu = data.coder_parameter.cpu.value
memory = "${data.coder_parameter.cpu.value}GiB"
}
device {
name = "home"
type = "disk"
properties = {
path = "/home/${local.workspace_user}"
pool = local.pool
source = incus_volume.home.name
}
}
device {
name = "docker"
type = "disk"
properties = {
path = "/var/lib/docker"
pool = local.pool
source = incus_volume.docker.name
source = incus_storage_volume.home.name
}
}
@@ -282,25 +318,23 @@ EOF
}
locals {
workspace_user = lower(data.coder_workspace_owner.me.name)
pool = "coder"
repo_base_dir = data.coder_parameter.repo_base_dir.value == "~" ? "/home/${local.workspace_user}" : replace(data.coder_parameter.repo_base_dir.value, "/^~\\//", "/home/${local.workspace_user}/")
repo_dir = module.git-clone.repo_dir
agent_id = data.coder_workspace.me.start_count == 1 ? coder_agent.main[0].id : ""
agent_token = data.coder_workspace.me.start_count == 1 ? coder_agent.main[0].token : ""
agent_init_script = data.coder_workspace.me.start_count == 1 ? coder_agent.main[0].init_script : ""
workspace_user = lower(data.coder_workspace_owner.me.name)
pool = data.coder_parameter.pool.value
# Workaround for the LXC provider stripping empty string config values, causing unexpected new values.
agent_token = data.coder_workspace.me.start_count == 1 ? coder_agent.main[0].token : "no-token"
agent_init_script = data.coder_workspace.me.start_count == 1 ? coder_agent.main[0].init_script : "#!/bin/sh\nexit 0"
}
resource "coder_metadata" "info" {
count = data.coder_workspace.me.start_count
resource_id = incus_instance.dev.name
resource_id = coder_agent.main[0].id
item {
key = "memory"
value = incus_instance.dev.limits.memory
value = incus_instance.dev.config["limits.memory"]
}
item {
key = "cpus"
value = incus_instance.dev.limits.cpu
value = incus_instance.dev.config["limits.cpu"]
}
item {
key = "instance"
@@ -308,10 +342,21 @@ resource "coder_metadata" "info" {
}
item {
key = "image"
value = "${incus_cached_image.image.source_remote}:${incus_cached_image.image.source_image}"
}
item {
key = "image_fingerprint"
value = substr(incus_cached_image.image.fingerprint, 0, 12)
value = data.coder_parameter.image.value
}
}
module "code-server" {
source = "registry.coder.com/coder/code-server/coder"
version = "~> 1.0"
agent_id = coder_agent.main[0].id
count = data.coder_workspace.me.start_count
}
module "git-clone" {
count = data.coder_workspace.me.start_count == 1 && data.coder_parameter.git_repo.value != "" ? 1 : 0
source = "registry.coder.com/coder/git-clone/coder"
version = "~> 1.0"
agent_id = coder_agent.main[0].id
url = data.coder_parameter.git_repo.value
}