chore(docs): add notes regarding provisioner tags and untagged jobs (#15081)

Relates to https://github.com/coder/coder/issues/15047

---------

Co-authored-by: Edward Angert <EdwardAngert@users.noreply.github.com>

docs/admin/provisioners.md

@@ -148,9 +148,19 @@ coder templates push on-prem-chicago \
   --provisioner-tag datacenter=chicago
 ```
 
+This can also be done in the UI when building a template:
+
+> ![template tags](../images/admin/provisioner-tags.png)
+
 Alternatively, a template can target a provisioner via
 [workspace tags](https://github.com/coder/coder/tree/main/examples/workspace-tags)
-inside the Terraform.
+inside the Terraform. See the
+[workspace tags documentation](../admin/templates/extending-templates/workspace-tags.md)
+for more information.
+
+> [!NOTE] Workspace tags defined with the `coder_workspace_tags` data source
+> template **do not** automatically apply to the template import job! You may
+> need to specify the desired tags when importing the template.
 
 A provisioner can run a given build job if one of the below is true:
 
@@ -170,6 +180,14 @@ However, it will not pick up any build jobs that do not have either of the
 `environment` or `datacenter` tags set. It will also not pick up any build jobs
 from templates with the tag `scope=user` set.
 
+> [!NOTE] If you only run tagged provisioners, you will need to specify a set of
+> tags that matches at least one provisioner for _all_ template import jobs and
+> workspace build jobs.
+>
+> You may wish to run at least one additional provisioner with no additional
+> tags so that provisioner jobs with no additional tags defined will be picked
+> up instead of potentially remaining in the Pending state indefinitely.
+
 This is illustrated in the below table:
 
 | Provisioner Tags                                                  | Job Tags                                                         | Can Run Job? |

docs/admin/templates/extending-templates/workspace-tags.md

@@ -17,6 +17,7 @@ specified:
 ```tf
 data "coder_workspace_tags" "custom_workspace_tags" {
   tags = {
+    "az"          = var.az
     "zone"        = "developers"
     "runtime"     = data.coder_parameter.runtime_selector.value
     "project_id"  = "PROJECT_${data.coder_parameter.project_name.value}"
@@ -50,6 +51,9 @@ added that can handle its combination of tags.
 Before releasing the template version with configurable workspace tags, ensure
 that every tag set is associated with at least one healthy provisioner.
 
+> [!NOTE] It may be useful to run at least one provisioner with no additional
+> tag restrictions that is able to take on any job.
+
 ### Parameters types
 
 Provisioners require job tags to be defined in plain string format. When a
@@ -66,6 +70,16 @@ the workspace owner to change a provisioner group (due to different tags). In
 most cases, `coder_parameter`s backing `coder_workspace_tags` should be marked
 as immutable and set only once, during workspace creation.
 
+We recommend using only the following as inputs for `coder_workspace_tags`:
+
+|                    | Example                                       |
+| :----------------- | :-------------------------------------------- |
+| Static values      | `"developers"`                                |
+| Template variables | `var.az`                                      |
+| Coder parameters   | `data.coder_parameter.runtime_selector.value` |
+
+Passing template tags in from other data sources may have undesired effects.
+
 ### HCL syntax
 
 When importing the template version with `coder_workspace_tags`, the Coder