From f5ccf68e534edd731ed54588d261f910cd259fbc Mon Sep 17 00:00:00 2001 From: Cian Johnston Date: Thu, 23 Apr 2026 08:24:03 +0100 Subject: [PATCH] feat: add lima incus example (#24640) Depends on https://github.com/coder/coder/pull/24616 Adds a sample Lima configuration for Coder+Incus. --- .../coder_templates_init_--help.golden | 2 +- docs/reference/cli/templates_init.md | 6 +- examples/examples.gen.json | 13 ++ examples/examples.go | 1 + examples/lima/README.md | 36 ++++- .../lima/{coder.yaml => coder-docker.yaml} | 6 +- examples/lima/coder-incus.yaml | 151 ++++++++++++++++++ 7 files changed, 201 insertions(+), 14 deletions(-) rename examples/lima/{coder.yaml => coder-docker.yaml} (96%) create mode 100644 examples/lima/coder-incus.yaml diff --git a/cli/testdata/coder_templates_init_--help.golden b/cli/testdata/coder_templates_init_--help.golden index 44be7a9529..dcf3f0e546 100644 --- a/cli/testdata/coder_templates_init_--help.golden +++ b/cli/testdata/coder_templates_init_--help.golden @@ -6,7 +6,7 @@ USAGE: Get started with a templated template. OPTIONS: - --id aws-devcontainer|aws-linux|aws-windows|azure-linux|digitalocean-linux|docker|docker-devcontainer|docker-envbuilder|gcp-devcontainer|gcp-linux|gcp-vm-container|gcp-windows|kubernetes|kubernetes-devcontainer|nomad-docker|scratch|tasks-docker + --id aws-devcontainer|aws-linux|aws-windows|azure-linux|digitalocean-linux|docker|docker-devcontainer|docker-envbuilder|gcp-devcontainer|gcp-linux|gcp-vm-container|gcp-windows|incus|kubernetes|kubernetes-devcontainer|nomad-docker|scratch|tasks-docker Specify a given example template by ID. ——— diff --git a/docs/reference/cli/templates_init.md b/docs/reference/cli/templates_init.md index 3ac28749ad..cf34de96bc 100644 --- a/docs/reference/cli/templates_init.md +++ b/docs/reference/cli/templates_init.md @@ -13,8 +13,8 @@ coder templates init [flags] [directory] ### --id -| | | -|------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| Type | aws-devcontainer\|aws-linux\|aws-windows\|azure-linux\|digitalocean-linux\|docker\|docker-devcontainer\|docker-envbuilder\|gcp-devcontainer\|gcp-linux\|gcp-vm-container\|gcp-windows\|kubernetes\|kubernetes-devcontainer\|nomad-docker\|scratch\|tasks-docker | +| | | +|------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Type | aws-devcontainer\|aws-linux\|aws-windows\|azure-linux\|digitalocean-linux\|docker\|docker-devcontainer\|docker-envbuilder\|gcp-devcontainer\|gcp-linux\|gcp-vm-container\|gcp-windows\|incus\|kubernetes\|kubernetes-devcontainer\|nomad-docker\|scratch\|tasks-docker | Specify a given example template by ID. diff --git a/examples/examples.gen.json b/examples/examples.gen.json index 5226d64cf6..05f82e439b 100644 --- a/examples/examples.gen.json +++ b/examples/examples.gen.json @@ -160,6 +160,19 @@ ], "markdown": "\n# Remote Development on Google Compute Engine (Windows)\n\n## Prerequisites\n\n### Authentication\n\nThis template assumes that coderd is run in an environment that is authenticated\nwith Google Cloud. For example, run `gcloud auth application-default login` to\nimport credentials on the system and user running coderd. For other ways to\nauthenticate [consult the Terraform\ndocs](https://registry.terraform.io/providers/hashicorp/google/latest/docs/guides/getting_started#adding-credentials).\n\nCoder requires a Google Cloud Service Account to provision workspaces. To create\na service account:\n\n1. Navigate to the [CGP\n console](https://console.cloud.google.com/projectselector/iam-admin/serviceaccounts/create),\n and select your Cloud project (if you have more than one project associated\n with your account)\n\n1. Provide a service account name (this name is used to generate the service\n account ID)\n\n1. Click **Create and continue**, and choose the following IAM roles to grant to\n the service account:\n\n - Compute Admin\n - Service Account User\n\n Click **Continue**.\n\n1. Click on the created key, and navigate to the **Keys** tab.\n\n1. Click **Add key** \u003e **Create new key**.\n\n1. Generate a **JSON private key**, which will be what you provide to Coder\n during the setup process.\n\n## Architecture\n\nThis template provisions the following resources:\n\n- GCP VM (ephemeral)\n- GCP Disk (persistent, mounted to root)\n\nCoder persists the root volume. The full filesystem is preserved when the workspace restarts. See this [community example](https://github.com/bpmct/coder-templates/tree/main/aws-linux-ephemeral) of an ephemeral AWS instance.\n\n\u003e **Note**\n\u003e This template is designed to be a starting point! Edit the Terraform to extend the template to support your use case.\n\n## code-server\n\n`code-server` is installed via the `startup_script` argument in the `coder_agent`\nresource block. The `coder_app` resource is defined to access `code-server` through\nthe dashboard UI over `localhost:13337`.\n" }, + { + "id": "incus", + "url": "", + "name": "Incus System Container with Docker", + "description": "Develop in an Incus System Container with Docker using Incus", + "icon": "/icon/lxc.svg", + "tags": [ + "incus", + "lxc", + "lxd" + ], + "markdown": "\n# Incus System Container with Docker\n\nDevelop in an Incus System Container and run nested Docker containers using Incus.\n\n## Architecture\n\nThis template uses the [Incus guest API](https://linuxcontainers.org/incus/docs/main/dev-incus/) (`/dev/incus/sock`) to deliver the Coder agent token and URL into the container without any host filesystem coupling. This means:\n\n- **The provisioner does not need to run on the Incus host.** There are no bind mounts or local file writes. All configuration is passed via Incus `user.*` config keys and read from inside the container at runtime.\n- **The agent binary is downloaded automatically.** The standard Coder init script fetches the correct binary from the Coder server on every boot, keeping it in sync with the server version.\n- **The agent token is refreshed on every start.** Terraform updates the `user.coder_agent_token` config key each workspace start. A watcher service inside the container listens for config changes via the guest API events endpoint and restarts the agent when a new token arrives.\n\n### Boot sequence\n\n1. **First boot (cloud-init):** Creates the workspace user, writes the bootstrap scripts and systemd units, installs `curl` and `git`, and enables the services. Cloud-init only runs once.\n2. **Every boot (systemd):**\n - `coder-agent-config.service` (oneshot) reads `CODER_AGENT_TOKEN` and `CODER_AGENT_URL` from the Incus guest API and writes them to `/opt/coder/init.env`.\n - `coder-agent.service` loads the env file and runs the Coder init script, which downloads the agent binary and starts it.\n - `coder-agent-watcher.service` streams config change events from the guest API. If the Incus provider updates the token *after* the container has already booted (a known provider ordering issue), the watcher detects the change, re-fetches the config, and restarts the agent.\n\n### Packages\n\nEssential packages (`curl`, `git`) are installed via cloud-init on first boot, before the agent starts. Additional packages (e.g. `docker.io`) are installed via a non-blocking [`coder_script`](https://registry.terraform.io/providers/coder/coder/latest/docs/resources/script) that runs on each workspace start. It does not block login; users can connect to the workspace immediately while packages install in the background. On subsequent starts, it detects packages are already installed and skips the installation.\n\n## Prerequisites\n\n1. Install [Incus](https://linuxcontainers.org/incus/) on a machine reachable by the Coder provisioner.\n2. Allow Coder to access the Incus socket.\n\n - If you're running Coder as a system service, run `sudo usermod -aG incus-admin coder` and restart the Coder service.\n - If you're running Coder as a Docker Compose service, get the group ID of the `incus-admin` group by running `getent group incus-admin` and add the following to your `compose.yaml` file:\n\n ```yaml\n services:\n coder:\n volumes:\n - /var/lib/incus/unix.socket:/var/lib/incus/unix.socket\n group_add:\n - 996 # Replace with the group ID of the `incus-admin` group\n ```\n\n3. Create a storage pool named `coder` by running `incus storage create coder btrfs` (or use another [supported driver](https://linuxcontainers.org/incus/docs/main/reference/storage_drivers/)).\n\n## Usage\n\n\u003e **Note:** This template requires a container image with cloud-init installed, such as `images:debian/13/cloud` or `images:ubuntu/24.04/cloud`. Images are pulled automatically from the [Linux Containers image server](https://images.linuxcontainers.org/).\n\n1. Run `coder templates push --directory .` from this directory.\n2. Create a workspace from the template in the Coder UI.\n\n## Parameters\n\n| Parameter | Description | Default |\n|--------------------|--------------------------------------------------------------------------------------------|--------------------------|\n| **Image** | Container image with cloud-init. Options: Debian 13, Debian 12, Ubuntu 24.04, Ubuntu 22.04 | `images:debian/13/cloud` |\n| **CPU** | Number of CPUs (1-8) | `1` |\n| **Memory** | Memory in GB (1-16) | `2` |\n| **Storage pool** | Incus storage pool name | `coder` |\n| **Git repository** | Clone a git repo inside the workspace | *(empty)* |\n\n## Extending this template\n\nSee the [lxc/incus](https://registry.terraform.io/providers/lxc/incus/latest/docs) Terraform provider documentation to add the following features to your Coder template:\n\n- Remote Incus hosts (HTTPS)\n- Additional volume mounts\n- Custom networks\n- GPU passthrough\n- More\n\nWe also welcome contributions!\n" + }, { "id": "kubernetes", "url": "", diff --git a/examples/examples.go b/examples/examples.go index 62cfb07782..c5b141bd0c 100644 --- a/examples/examples.go +++ b/examples/examples.go @@ -36,6 +36,7 @@ var ( //go:embed templates/gcp-linux //go:embed templates/gcp-vm-container //go:embed templates/gcp-windows + //go:embed templates/incus //go:embed templates/kubernetes //go:embed templates/kubernetes-devcontainer //go:embed templates/nomad-docker diff --git a/examples/lima/README.md b/examples/lima/README.md index aac38a8ec2..565bc34422 100644 --- a/examples/lima/README.md +++ b/examples/lima/README.md @@ -1,19 +1,22 @@ --- name: Run Coder in Lima description: Quickly stand up Coder using Lima -tags: [local, docker, vm, lima] +tags: [local, docker, incus, vm, lima] --- # Run Coder in Lima -This provides a sample [Lima](https://github.com/lima-vm/lima) configuration for Coder. +This provides sample [Lima](https://github.com/lima-vm/lima) configurations for Coder. This lets you quickly test out Coder in a self-contained environment. +The Docker configuration runs workspaces in Docker containers; the Incus configuration runs workspaces in Incus system containers (with Docker available inside each workspace). > Prerequisite: You must have `lima` installed and available to use this. -## Getting Started +## Getting Started (Docker) -- Run `limactl start --name=coder https://raw.githubusercontent.com/coder/coder/main/examples/lima/coder.yaml` +This configuration (`coder-docker.yaml`) creates a VM to run Coder workspaces in Docker. + +- Run `limactl start --name=coder https://raw.githubusercontent.com/coder/coder/main/examples/lima/coder-docker.yaml` - You can use the configuration as-is, or edit it to your liking. This will: @@ -21,13 +24,32 @@ This will: - Start an Ubuntu 22.04 VM - Install Docker and Terraform from the official repos - Install Coder using the [installation script](../../docs/install/install.sh.md) -- Generates an initial user account `admin@coder.com` with a randomly generated password (stored in the VM under `/home/${USER}.linux/.config/coderv2/password`) -- Initializes a [sample Docker template](https://github.com/coder/coder/tree/main/examples/templates/docker) for creating workspaces +- Generate an initial user account `admin@coder.com` with a randomly generated password (stored in the VM under `/home/${USER}.linux/.config/coderv2/password`) +- Initialize a [sample Docker template](https://github.com/coder/coder/tree/main/examples/templates/docker) for creating workspaces Once this completes, you can visit `http://localhost:3000` and start creating workspaces! Alternatively, enter the VM with `limactl shell coder` and run `coder templates init` to start creating your own templates! +## Getting Started (Incus) + +This configuration (`coder-incus.yaml`) creates a VM to run Coder workspaces in Incus. + +- Run `limactl start --name=coder-incus https://raw.githubusercontent.com/coder/coder/main/examples/lima/coder-incus.yaml` +- You can use the configuration as-is, or edit it to your liking. + +This will: + +- Start a Debian 13 VM +- Install Incus from the Debian repos and Terraform via the Coder installer +- Install Coder using the [installation script](../../docs/install/install.sh.md) +- Generate an initial user account `admin@coder.com` with a randomly generated password (stored in the VM under `/home/${USER}.linux/.config/coderv2/password`) +- Initialize a [sample Incus template](https://github.com/coder/coder/tree/main/examples/templates/incus) for creating workspaces + +Once this completes, you can visit `http://localhost:3000` and start creating workspaces! + +Alternatively, enter the VM with `limactl shell coder-incus` and run `coder templates init` to start creating your own templates! + ## Further Information -- To learn more about Lima, [visit the the project's GitHub page](https://github.com/lima-vm/lima/). +- To learn more about Lima, [visit the project's GitHub page](https://github.com/lima-vm/lima/). diff --git a/examples/lima/coder.yaml b/examples/lima/coder-docker.yaml similarity index 96% rename from examples/lima/coder.yaml rename to examples/lima/coder-docker.yaml index 1d7358ccdf..a6e2e0f7ec 100644 --- a/examples/lima/coder.yaml +++ b/examples/lima/coder-docker.yaml @@ -1,8 +1,8 @@ -# Deploy Coder in Lima via the install script +# Deploy Coder in Lima with Docker via the install script # See: https://coder.com/docs/install -# $ limactl start ./coder.yaml +# $ limactl start ./coder-docker.yaml # $ limactl shell coder -# The web UI is accessible on http://localhost:3000 -- ports are forwarded automatically by lima: +# The web UI is accessible on http://localhost:3000. Ports are forwarded automatically by Lima. # $ coder login http://localhost:3000 # This example requires Lima v0.8.3 or later. diff --git a/examples/lima/coder-incus.yaml b/examples/lima/coder-incus.yaml new file mode 100644 index 0000000000..4ba9abf563 --- /dev/null +++ b/examples/lima/coder-incus.yaml @@ -0,0 +1,151 @@ +# Deploy Coder in Lima with Incus +# See: https://coder.com/docs/install +# $ limactl start ./coder-incus.yaml +# $ limactl shell coder-incus +# The web UI is accessible on http://localhost:3000. Ports are forwarded automatically by Lima. +# $ coder login http://localhost:3000 + +minimumLimaVersion: "2.0.0" + +images: + - location: "https://cloud.debian.org/images/cloud/trixie/20260327-2429/debian-13-genericcloud-amd64-20260327-2429.qcow2" + arch: "x86_64" + digest: "sha512:09559ec27d263997827dd8cddf76e97ea8e0f1803380aa501ea7eaa4b4968cd76ffef4ec7eb07ef1a9ccbeb0925a5020492ea9ed53eb167d62f3a2285039912c" + - location: "https://cloud.debian.org/images/cloud/trixie/20260327-2429/debian-13-genericcloud-arm64-20260327-2429.qcow2" + arch: "aarch64" + digest: "sha512:cb25e88240d8760c860f780c42257472f7c63c1ab54368c4eaa4ddb44e1e6224df8e719ee7ab0fb0d52d5de505f98034dd44ee73a9d9dcf66a2035215f1e8512" + # Fallback to the latest release image. + # Hint: run `limactl prune` to invalidate the cache + - location: "https://cloud.debian.org/images/cloud/trixie/daily/latest/debian-13-genericcloud-amd64-daily.qcow2" + arch: "x86_64" + - location: "https://cloud.debian.org/images/cloud/trixie/daily/latest/debian-13-genericcloud-arm64-daily.qcow2" + arch: "aarch64" + +# Disable 9p mounts; they are not supported by the Debian cloud image kernel. +mountTypesUnsupported: [9p] + +# Your home directory is mounted read-only +mounts: + - location: "~" +containerd: + system: false + user: false +provision: + - mode: system + script: | + #!/bin/bash + set -eux -o pipefail + command -v incus >/dev/null 2>&1 && exit 0 + export DEBIAN_FRONTEND=noninteractive + # Wait for any apt locks from unattended-upgrades on first boot + while fuser /var/lib/dpkg/lock-frontend >/dev/null 2>&1; do sleep 1; done + # Incus is available natively in Debian Trixie + apt-get update + apt-get install -qqy incus btrfs-progs + # Initialize Incus with preseed config. + # We use an explicit subnet because --minimal's auto-detection fails + # when Lima's own bridge already claims the common ranges. + cat <<'PRESEED' | incus admin init --preseed + networks: + - name: incusbr0 + type: bridge + config: + ipv4.address: 10.155.0.1/24 + ipv4.nat: "true" + ipv6.address: none + storage_pools: + - name: coder + driver: btrfs + profiles: + - name: default + devices: + eth0: + name: eth0 + network: incusbr0 + type: nic + root: + path: / + pool: coder + type: disk + PRESEED + # Give the Lima user access to Incus + usermod -aG incus-admin {{.User}} + - mode: system + script: | + #!/bin/bash + set -eux -o pipefail + command -v coder >/dev/null 2>&1 && exit 0 + export DEBIAN_FRONTEND=noninteractive + export HOME=/root + # Wait for any apt locks from unattended-upgrades on first boot + while fuser /var/lib/dpkg/lock-frontend >/dev/null 2>&1; do sleep 1; done + # Using install.sh --with-terraform requires unzip to be available. + apt-get update + apt-get install -qqy unzip + curl -fsSL https://coder.com/install.sh | sh -s -- --with-terraform + # Ensure Coder has access to the Incus socket + usermod -aG incus-admin coder + # Ensure coder listens on all interfaces + sed -i 's/CODER_HTTP_ADDRESS=.*/CODER_HTTP_ADDRESS=0.0.0.0:3000/' /etc/coder.d/coder.env + # Also set the access URL to host.lima.internal for fast deployments + sed -i 's#CODER_ACCESS_URL=.*#CODER_ACCESS_URL=http://host.lima.internal:3000#' /etc/coder.d/coder.env + # Ensure coder starts on boot + systemctl enable coder + systemctl start coder + # Wait for Terraform to be installed + timeout 60s bash -c 'until /usr/local/bin/terraform version >/dev/null 2>&1; do sleep 1; done' + - mode: user + script: | + #!/bin/bash + set -eux -o pipefail + # If we are already logged in, nothing to do + coder templates list >/dev/null 2>&1 && exit 0 + # Set up initial user + [ ! -e ~/.config/coderv2/session ] && coder login http://localhost:3000 \ + --first-user-username admin \ + --first-user-email admin@coder.com \ + --first-user-password "$(< /dev/urandom tr -dc _A-Z-a-z-0-9 | head -c12 | tee ${HOME}/.config/coderv2/password)" + # Create an initial Incus template + coder templates init --id incus + pushd ./incus + coder templates push incus --yes + popd + rm -rf ./incus +probes: + - description: "incus to be installed" + script: | + #!/bin/bash + set -eux -o pipefail + if ! timeout 30s bash -c "until command -v incus >/dev/null 2>&1; do sleep 3; done"; then + echo >&2 "incus is not installed yet" + exit 1 + fi + hint: | + See `/var/log/lima-guestagent.log` or run `limactl shell coder-incus` to debug. + - description: "coder to be installed" + script: | + #!/bin/bash + set -eux -o pipefail + if ! timeout 30s bash -c "until command -v coder >/dev/null 2>&1; do sleep 3; done"; then + echo >&2 "coder is not installed yet" + exit 1 + fi + hint: | + See `/var/log/lima-guestagent.log` or run `limactl shell coder-incus` to debug. +message: | + All Done! Your Coder instance is accessible at http://localhost:3000 + + Username: "admin@coder.com" + Password: Run `LIMA_INSTANCE={{.Instance.Name}} lima cat /home/${USER}.linux/.config/coderv2/password` + + Create your first workspace: + ------ + limactl shell {{.Instance.Name}} + coder create my-workspace --template incus + ------ + + Get started creating your own template now: + ------ + limactl shell {{.Instance.Name}} + cd && coder templates init + ------