diff --git a/examples/templates/incus/README.md b/examples/templates/incus/README.md index 2300e6573f..603ba76456 100644 --- a/examples/templates/incus/README.md +++ b/examples/templates/incus/README.md @@ -1,22 +1,42 @@ --- display_name: Incus System Container with Docker -description: Develop in an Incus System Container with Docker using incus +description: Develop in an Incus System Container with Docker using Incus icon: ../../../site/static/icon/lxc.svg maintainer_github: coder verified: true -tags: [local, incus, lxc, lxd] +tags: [incus, lxc, lxd] --- # Incus System Container with Docker -Develop in an Incus System Container and run nested Docker containers using Incus on your local infrastructure. +Develop in an Incus System Container and run nested Docker containers using Incus. + +## Architecture + +This template uses the [Incus guest API](https://linuxcontainers.org/incus/docs/main/dev-incus/) (`/dev/incus/sock`) to deliver the Coder agent token and URL into the container without any host filesystem coupling. This means: + +- **The provisioner does not need to run on the Incus host.** There are no bind mounts or local file writes. All configuration is passed via Incus `user.*` config keys and read from inside the container at runtime. +- **The agent binary is downloaded automatically.** The standard Coder init script fetches the correct binary from the Coder server on every boot, keeping it in sync with the server version. +- **The agent token is refreshed on every start.** Terraform updates the `user.coder_agent_token` config key each workspace start. A watcher service inside the container listens for config changes via the guest API events endpoint and restarts the agent when a new token arrives. + +### Boot sequence + +1. **First boot (cloud-init):** Creates the workspace user, writes the bootstrap scripts and systemd units, installs `curl` and `git`, and enables the services. Cloud-init only runs once. +2. **Every boot (systemd):** + - `coder-agent-config.service` (oneshot) reads `CODER_AGENT_TOKEN` and `CODER_AGENT_URL` from the Incus guest API and writes them to `/opt/coder/init.env`. + - `coder-agent.service` loads the env file and runs the Coder init script, which downloads the agent binary and starts it. + - `coder-agent-watcher.service` streams config change events from the guest API. If the Incus provider updates the token *after* the container has already booted (a known provider ordering issue), the watcher detects the change, re-fetches the config, and restarts the agent. + +### Packages + +Essential packages (`curl`, `git`) are installed via cloud-init on first boot, before the agent starts. Additional packages (e.g. `docker.io`) are installed via a non-blocking [`coder_script`](https://registry.terraform.io/providers/coder/coder/latest/docs/resources/script) that runs on each workspace start. It does not block login; users can connect to the workspace immediately while packages install in the background. On subsequent starts, it detects packages are already installed and skips the installation. ## Prerequisites -1. Install [Incus](https://linuxcontainers.org/incus/) on the same machine as Coder. +1. Install [Incus](https://linuxcontainers.org/incus/) on a machine reachable by the Coder provisioner. 2. Allow Coder to access the Incus socket. - - If you're running Coder as system service, run `sudo usermod -aG incus-admin coder` and restart the Coder service. + - If you're running Coder as a system service, run `sudo usermod -aG incus-admin coder` and restart the Coder service. - If you're running Coder as a Docker Compose service, get the group ID of the `incus-admin` group by running `getent group incus-admin` and add the following to your `compose.yaml` file: ```yaml @@ -28,24 +48,33 @@ Develop in an Incus System Container and run nested Docker containers using Incu - 996 # Replace with the group ID of the `incus-admin` group ``` -3. Create a storage pool named `coder` and `btrfs` as the driver by running `incus storage create coder btrfs`. +3. Create a storage pool named `coder` by running `incus storage create coder btrfs` (or use another [supported driver](https://linuxcontainers.org/incus/docs/main/reference/storage_drivers/)). ## Usage -> **Note:** this template requires using a container image with cloud-init installed such as `ubuntu/jammy/cloud/amd64`. +> **Note:** This template requires a container image with cloud-init installed, such as `images:debian/13/cloud` or `images:ubuntu/24.04/cloud`. Images are pulled automatically from the [Linux Containers image server](https://images.linuxcontainers.org/). -1. Run `coder templates init -id incus` -1. Select this template -1. Follow the on-screen instructions +1. Run `coder templates push --directory .` from this directory. +2. Create a workspace from the template in the Coder UI. + +## Parameters + +| Parameter | Description | Default | +|--------------------|--------------------------------------------------------------------------------------------|--------------------------| +| **Image** | Container image with cloud-init. Options: Debian 13, Debian 12, Ubuntu 24.04, Ubuntu 22.04 | `images:debian/13/cloud` | +| **CPU** | Number of CPUs (1-8) | `1` | +| **Memory** | Memory in GB (1-16) | `2` | +| **Storage pool** | Incus storage pool name | `coder` | +| **Git repository** | Clone a git repo inside the workspace | *(empty)* | ## Extending this template -See the [lxc/incus](https://registry.terraform.io/providers/lxc/incus/latest/docs) Terraform provider documentation to -add the following features to your Coder template: +See the [lxc/incus](https://registry.terraform.io/providers/lxc/incus/latest/docs) Terraform provider documentation to add the following features to your Coder template: -- HTTPS incus host -- Volume mounts +- Remote Incus hosts (HTTPS) +- Additional volume mounts - Custom networks +- GPU passthrough - More We also welcome contributions! diff --git a/examples/templates/incus/main.tf b/examples/templates/incus/main.tf index 95e10a6d2b..d8d8551549 100644 --- a/examples/templates/incus/main.tf +++ b/examples/templates/incus/main.tf @@ -1,10 +1,12 @@ terraform { required_providers { coder = { - source = "coder/coder" + source = "coder/coder" + version = "~>2" } incus = { - source = "lxc/incus" + source = "lxc/incus" + version = "~>1.0" } } } @@ -19,10 +21,28 @@ data "coder_workspace_owner" "me" {} data "coder_parameter" "image" { name = "image" display_name = "Image" - description = "The container image to use. Be sure to use a variant with cloud-init installed!" - default = "ubuntu/jammy/cloud/amd64" + description = "The container image to use. Must have cloud-init installed." + default = "images:debian/13/cloud" icon = "/icon/image.svg" - mutable = true + mutable = false + + option { + name = "Debian 13 (Trixie)" + value = "images:debian/13/cloud" + } + option { + name = "Debian 12 (Bookworm)" + value = "images:debian/12/cloud" + } + option { + name = "Ubuntu 24.04 (Noble)" + value = "images:ubuntu/24.04/cloud" + } + option { + name = "Ubuntu 22.04 (Jammy)" + value = "images:ubuntu/22.04/cloud" + } + } data "coder_parameter" "cpu" { @@ -56,17 +76,18 @@ data "coder_parameter" "memory" { data "coder_parameter" "git_repo" { type = "string" name = "Git repository" - default = "https://github.com/coder/coder" - description = "Clone a git repo into [base directory]" + default = "" + description = "Clone a git repo inside the workspace" mutable = true } -data "coder_parameter" "repo_base_dir" { - type = "string" - name = "Repository Base Directory" - default = "~" - description = "The directory specified will be created (if missing) and the specified repo will be cloned into [base directory]/{repo}🪄." - mutable = true +data "coder_parameter" "pool" { + type = "string" + name = "pool" + display_name = "Storage pool" + default = "coder" + description = "Incus storage pool name" + mutable = false } resource "coder_agent" "main" { @@ -75,7 +96,9 @@ resource "coder_agent" "main" { os = "linux" dir = "/home/${local.workspace_user}" env = { - CODER_WORKSPACE_ID = data.coder_workspace.me.id + CODER_WORKSPACE_ID = data.coder_workspace.me.id + CODER_SESSION_TOKEN = data.coder_workspace_owner.me.session_token + CODER_URL = data.coder_workspace.me.access_url } metadata { @@ -93,87 +116,74 @@ resource "coder_agent" "main" { interval = 10 timeout = 1 } - - metadata { - display_name = "Home Disk" - key = "3_home_disk" - script = "coder stat disk --path /home/${lower(data.coder_workspace_owner.me.name)}" - interval = 60 - timeout = 1 - } } -# https://registry.coder.com/modules/coder/git-clone -module "git-clone" { - source = "registry.coder.com/coder/git-clone/coder" - # This ensures that the latest non-breaking version of the module gets downloaded, you can also pin the module version to prevent breaking changes in production. - version = "~> 1.0" - agent_id = local.agent_id - url = data.coder_parameter.git_repo.value - base_dir = local.repo_base_dir +# Note: execution order is currently not guaranteed so only +# include packages here that are not required for either the +# agent or modules. +resource "coder_script" "packages" { + count = data.coder_workspace.me.start_count + agent_id = coder_agent.main[0].id + display_name = "Install packages" + icon = "/icon/debian.svg" + run_on_start = true + script = <<-EOF + #!/bin/bash + set -e + PACKAGES=(docker.io) + MISSING=() + for pkg in "$${PACKAGES[@]}"; do + if ! dpkg -s "$pkg" &> /dev/null; then + MISSING+=("$pkg") + fi + done + if [ "$${#MISSING[@]}" -gt 0 ]; then + echo "Installing: $${MISSING[*]}" + sudo apt-get update + sudo apt-get install -y "$${MISSING[@]}" + + echo "Packages installed successfully" + else + echo "All packages already installed" + fi + # Ensure the workspace user can access the Docker socket without + # needing the docker group (which would require a new login session). + if [ -S /var/run/docker.sock ]; then + sudo chown $(whoami) /var/run/docker.sock + fi + EOF } -# https://registry.coder.com/modules/coder/code-server -module "code-server" { - source = "registry.coder.com/coder/code-server/coder" - # This ensures that the latest non-breaking version of the module gets downloaded, you can also pin the module version to prevent breaking changes in production. - version = "~> 1.0" - agent_id = local.agent_id - folder = local.repo_base_dir -} - -# https://registry.coder.com/modules/coder/filebrowser -module "filebrowser" { - source = "registry.coder.com/coder/filebrowser/coder" - # This ensures that the latest non-breaking version of the module gets downloaded, you can also pin the module version to prevent breaking changes in production. - version = "~> 1.0" - agent_id = local.agent_id -} - -# https://registry.coder.com/modules/coder/coder-login -module "coder-login" { - source = "registry.coder.com/coder/coder-login/coder" - # This ensures that the latest non-breaking version of the module gets downloaded, you can also pin the module version to prevent breaking changes in production. - version = "~> 1.0" - agent_id = local.agent_id -} - -resource "incus_volume" "home" { +resource "incus_storage_volume" "home" { name = "coder-${data.coder_workspace.me.id}-home" pool = local.pool } -resource "incus_volume" "docker" { - name = "coder-${data.coder_workspace.me.id}-docker" - pool = local.pool -} - -resource "incus_cached_image" "image" { - source_remote = "images" - source_image = data.coder_parameter.image.value -} - -resource "incus_instance_file" "agent_token" { - count = data.coder_workspace.me.start_count - instance = incus_instance.dev.name - content = < /opt/coder/init.env + # The standard Coder agent init script, provided by coder_agent.init_script. + # This handles downloading the correct agent binary and running it. + - path: /opt/coder/coder-init.sh permissions: "0755" encoding: b64 content: ${base64encode(local.agent_init_script)} + - path: /etc/systemd/system/coder-agent-config.service + permissions: "0644" + content: | + [Unit] + Description=Fetch Coder Agent Config from Incus Guest API + After=network-online.target + Wants=network-online.target + + [Service] + Type=oneshot + ExecStart=/opt/coder/fetch-config.sh + # Watcher script that listens for config changes via the Incus guest API + # events endpoint. The Incus Terraform provider starts the instance before + # updating config keys, so on a stop->start cycle the agent initially boots + # with a stale token. This watcher detects when user.coder_agent_token is + # updated, re-fetches the config, and restarts the agent with the new token. + - path: /opt/coder/watch-config.sh + permissions: "0755" + content: | + #!/bin/bash + INCUS_SOCK="/dev/incus/sock" + curl -sfN --unix-socket "$INCUS_SOCK" http://localhost/1.0/events?type=config | \ + while read -r event; do + key=$(echo "$event" | sed -n 's/.*"key":"\([^"]*\)".*/\1/p') + if [ "$key" = "user.coder_agent_token" ]; then + /opt/coder/fetch-config.sh + systemctl restart coder-agent.service + fi + done + - path: /etc/systemd/system/coder-agent-watcher.service + permissions: "0644" + content: | + [Unit] + Description=Watch for Coder Agent config changes via Incus Guest API + After=network-online.target + Wants=network-online.target + + [Service] + ExecStart=/opt/coder/watch-config.sh + Restart=always + RestartSec=5 + + [Install] + WantedBy=multi-user.target - path: /etc/systemd/system/coder-agent.service permissions: "0644" content: | [Unit] Description=Coder Agent - After=network-online.target + After=network-online.target coder-agent-config.service Wants=network-online.target + Requires=coder-agent-config.service [Service] User=${local.workspace_user} EnvironmentFile=/opt/coder/init.env - ExecStart=/opt/coder/init + ExecStart=/opt/coder/coder-init.sh Restart=always RestartSec=10 TimeoutStopSec=90 KillMode=process - OOMScoreAdjust=-900 SyslogIdentifier=coder-agent - [Install] - WantedBy=multi-user.target - - path: /etc/systemd/system/coder-agent-watcher.service - permissions: "0644" - content: | - [Unit] - Description=Coder Agent Watcher - After=network-online.target - - [Service] - Type=oneshot - ExecStart=/usr/bin/systemctl restart coder-agent.service - - [Install] - WantedBy=multi-user.target - - path: /etc/systemd/system/coder-agent-watcher.path - permissions: "0644" - content: | - [Path] - PathModified=/opt/coder/init.env - Unit=coder-agent-watcher.service - [Install] WantedBy=multi-user.target runcmd: - chown -R ${local.workspace_user}:${local.workspace_user} /home/${local.workspace_user} - - | - #!/bin/bash - apt-get update && apt-get install -y curl docker.io - usermod -aG docker ${local.workspace_user} - newgrp docker - - systemctl enable coder-agent.service coder-agent-watcher.service coder-agent-watcher.path - - systemctl start coder-agent.service coder-agent-watcher.service coder-agent-watcher.path + # Install package dependencies before starting the agent. + - apt-get update && apt-get install -y curl git + - systemctl daemon-reload + - systemctl enable coder-agent.service coder-agent-watcher.service + - systemctl start coder-agent.service coder-agent-watcher.service EOF } - limits = { - cpu = data.coder_parameter.cpu.value - memory = "${data.coder_parameter.cpu.value}GiB" - } - device { name = "home" type = "disk" properties = { path = "/home/${local.workspace_user}" pool = local.pool - source = incus_volume.home.name - } - } - - device { - name = "docker" - type = "disk" - properties = { - path = "/var/lib/docker" - pool = local.pool - source = incus_volume.docker.name + source = incus_storage_volume.home.name } } @@ -282,25 +318,23 @@ EOF } locals { - workspace_user = lower(data.coder_workspace_owner.me.name) - pool = "coder" - repo_base_dir = data.coder_parameter.repo_base_dir.value == "~" ? "/home/${local.workspace_user}" : replace(data.coder_parameter.repo_base_dir.value, "/^~\\//", "/home/${local.workspace_user}/") - repo_dir = module.git-clone.repo_dir - agent_id = data.coder_workspace.me.start_count == 1 ? coder_agent.main[0].id : "" - agent_token = data.coder_workspace.me.start_count == 1 ? coder_agent.main[0].token : "" - agent_init_script = data.coder_workspace.me.start_count == 1 ? coder_agent.main[0].init_script : "" + workspace_user = lower(data.coder_workspace_owner.me.name) + pool = data.coder_parameter.pool.value + # Workaround for the LXC provider stripping empty string config values, causing unexpected new values. + agent_token = data.coder_workspace.me.start_count == 1 ? coder_agent.main[0].token : "no-token" + agent_init_script = data.coder_workspace.me.start_count == 1 ? coder_agent.main[0].init_script : "#!/bin/sh\nexit 0" } resource "coder_metadata" "info" { count = data.coder_workspace.me.start_count - resource_id = incus_instance.dev.name + resource_id = coder_agent.main[0].id item { key = "memory" - value = incus_instance.dev.limits.memory + value = incus_instance.dev.config["limits.memory"] } item { key = "cpus" - value = incus_instance.dev.limits.cpu + value = incus_instance.dev.config["limits.cpu"] } item { key = "instance" @@ -308,10 +342,21 @@ resource "coder_metadata" "info" { } item { key = "image" - value = "${incus_cached_image.image.source_remote}:${incus_cached_image.image.source_image}" - } - item { - key = "image_fingerprint" - value = substr(incus_cached_image.image.fingerprint, 0, 12) + value = data.coder_parameter.image.value } } + +module "code-server" { + source = "registry.coder.com/coder/code-server/coder" + version = "~> 1.0" + agent_id = coder_agent.main[0].id + count = data.coder_workspace.me.start_count +} + +module "git-clone" { + count = data.coder_workspace.me.start_count == 1 && data.coder_parameter.git_repo.value != "" ? 1 : 0 + source = "registry.coder.com/coder/git-clone/coder" + version = "~> 1.0" + agent_id = coder_agent.main[0].id + url = data.coder_parameter.git_repo.value +}