---
sidebar_position: 2
---
# Test Definitions

Test definitions are reusable commands that appear in the Testing menu and can be run as standalone test tasks. They let teams standardize common checks such as unit tests, linting, browser smoke tests, IBM i compile checks, or targeted test-file runs.

Use test definitions when the command should be available to every user of an environment. Use a one-off command when you only need to run something once.

## Where Tests Live

Tests are stored in the environment's `tests.json` file in the coder setup repository:

```text
environments/
  my-environment/
    environment.json
    tests.json
```

The file is a JSON object. Each key is the test name users see in the UI and CLI.

```json
{
  "unit": {
    "description": "Run the unit test suite",
    "command": "npm test"
  },
  "lint-and-types": {
    "description": "Run linting and TypeScript checks",
    "commands": [
      "npm run lint",
      "npm run typecheck"
    ]
  }
}
```

If `tests.json` is missing, the environment simply has no named test commands.

## Schema

Each test definition can include:

| Field | Required | Purpose |
| --- | --- | --- |
| `description` | No | Text shown in the Testing menu and `coder test --list`. |
| `command` | Yes, unless `commands` is present | Single shell command to run. |
| `commands` | Yes, unless `command` is present | Array of commands joined with `&&`. |
| `parameters` | No | Prompted values substituted into the command. |
| `repos` | No | CLI local-state filter. Omit for all repos, list repo names for selected repos, or use `[]` to skip local state. |

Commands run in an ephemeral test container for the selected environment. Test tasks are visible in CoderFlow, stream output, and finish with the command's exit code.

## Parameters

Parameters let users customize a test before it runs. Use `${parameter_name}` placeholders in the command or commands:

```json
{
  "single-file": {
    "description": "Run one test file",
    "command": "npm test -- ${test_file}",
    "parameters": {
      "test_file": {
        "type": "files",
        "description": "Test file",
        "required": true,
        "prompt": {
          "type": "files",
          "label": "Choose a test file",
          "path": "/workspace/app/tests",
          "pattern": "*.test.js"
        }
      }
    }
  }
}
```

Common parameter fields:

| Field | Purpose |
| --- | --- |
| `type` | The parameter type. Use `text`, `list`, `command`, or `files` for prompted inputs. Boolean values can be authored, but the test run dialog passes them as normal substituted values. |
| `label` | Short display label. |
| `description` | Help text shown near the input. |
| `required` | Blocks the run until a value is provided. |
| `default` | Default value. |
| `allowCustomValue` | Set to `false` when users must pick from the prompt choices. |
| `prompt` | Prompt configuration for lists, dynamic commands, and file browsers. |

The Web UI collects parameter values before submitting the test. The server validates required parameters and substitutes `${name}` placeholders before running the command.

## Prompt Types

Use a list prompt when choices are known ahead of time:

```json
{
  "parameters": {
    "mode": {
      "type": "list",
      "required": true,
      "prompt": {
        "type": "list",
        "label": "Mode",
        "options": [
          { "value": "fast", "label": "Fast" },
          { "value": "coverage", "label": "Coverage" }
        ]
      }
    }
  }
}
```

Use a command prompt when choices should be discovered from the repository or environment:

```json
{
  "parameters": {
    "package": {
      "type": "command",
      "required": true,
      "prompt": {
        "type": "command",
        "label": "Package",
        "command": "find packages -maxdepth 1 -mindepth 1 -type d -printf '%f\\n' | sort",
        "parser": "lines"
      }
    }
  }
}
```

Use a files prompt to browse files or directories:

```json
{
  "parameters": {
    "source_member": {
      "type": "files",
      "required": true,
      "prompt": {
        "type": "files",
        "label": "Source member",
        "path": "/workspace/app/QRPGLESRC",
        "pattern": "*.{rpgle,sqlrpgle}",
        "include": "files"
      }
    }
  }
}
```

Dynamic prompt commands can use parsers:

| Parser | Expected output |
| --- | --- |
| `json` | JSON array of strings or `{ "value", "label", "description" }` objects. |
| `lines` | One choice per non-empty line. |
| `git-branches` | Output from `git branch` or `git branch -r`. |
| `files` | File paths, displayed by basename. |

Prompt commands run in a temporary environment container. Keep them read-only and fast.

## Multiple Selections

Set `prompt.multiSelect` to `true` when users can pick more than one value:

```json
{
  "parameters": {
    "test_file": {
      "type": "files",
      "required": true,
      "prompt": {
        "type": "files",
        "label": "Test files",
        "path": "/workspace/app/tests",
        "pattern": "*.test.js",
        "multiSelect": true
      }
    }
  }
}
```

When a multi-select parameter has several selected values, CoderFlow expands the test into combinations and runs each substituted command in sequence inside one test task. If multiple multi-select parameters are used, CoderFlow runs the cartesian product of the selected values. The server rejects runs with more than 100 combinations.

## Repository Local State

The CLI uses `repos` to decide which local repository changes to capture before running a test:

```json
{
  "unit": {
    "description": "Run app unit tests",
    "command": "npm test",
    "repos": ["app"]
  },
  "env-info": {
    "description": "Print environment details without syncing local changes",
    "command": "node --version && npm --version",
    "repos": []
  }
}
```

Behavior:

- Omit `repos` to capture local state for all configured environment repositories.
- Set `repos` to selected repository names to capture only those repos.
- Set `repos` to an empty array to skip local state capture.

The Testing menu starts test tasks from the server-side environment state. The CLI local-state behavior applies to `coder test`.

## Running From The Testing Menu

Users run configured tests from a task page:

1. Open the **Testing** menu.
2. Choose one or more test commands.
3. Fill in any required parameters.
4. Click **Run**.
5. Review the streamed output and exit code.

See [Testing Menu](/testing/testing-menu) for the runtime flow.

## Running From The CLI

Use `coder test --list` to inspect named tests:

```bash
coder test --list --env=my-environment
```

Run a named test:

```bash
coder test unit --env=my-environment
```

Run a one-off command instead of a named definition:

```bash
coder test --cmd="npm test -- --runInBand" --env=my-environment
```

`coder test` supports `--env=<environment>`, `--environment=<environment>`, `--cmd="<command>"`, `--list`, and `--no-local-state`.

The CLI lists and runs named test definitions, but it does not currently provide an option for passing named parameter values to tests. Run parameterized tests from the Web UI, or call the test-task API with `test_parameters`.

## Authoring Workflow

1. Open the environment's **Tests** tab or edit `tests.json` directly.
2. Add a short name, description, and command.
3. Use `commands` when the test has ordered steps that should stop on the first failure.
4. Add parameters for values users should choose at run time.
5. Add `repos` when CLI local-state capture should be limited.
6. Save, commit, and push the setup repo change.
7. Confirm the test appears in the Testing menu and in `coder test --list`.
