# @elench/testkit

`@elench/testkit` discovers `*.testkit.ts` files, infers suite ownership from the
filesystem, starts local services, provisions Docker-managed local Postgres
databases, and runs HTTP, DAL, and Playwright suites.

The package is now driven by `testkit.setup.ts`, not `testkit.config.json`.

## Usage

```bash
cd my-product

# Run every testkit-managed suite
npx @elench/testkit

# Inspect discovered tests without running them
npx @elench/testkit discover
npx @elench/testkit discover --output-mode verbose
npx @elench/testkit discover --json > .testkit/discovery.json
npx @elench/testkit discover --output .testkit/discovery.json

# Filter by type
npx @elench/testkit --type int
npx @elench/testkit --type dal
npx @elench/testkit --type e2e
npx @elench/testkit --type int,e2e,dal
npx @elench/testkit --type pw

# Parallel file execution
npx @elench/testkit --workers 8

# One file-level wall clock budget for every suite file
npx @elench/testkit --file-timeout-seconds 60

# Run a deterministic shard
npx @elench/testkit --shard 1/3

# Specific service / suite
npx @elench/testkit --service frontend --type pw -s navigation
npx @elench/testkit --service api --type int -s health
npx @elench/testkit --type int,e2e,dal -s dal:queries

# Exact file
npx @elench/testkit --type int --file __testkit__/health/health.int.testkit.ts

# Temporarily ignore repo-declared skip rules
npx @elench/testkit --ignore-skip-rules --file __testkit__/billing/billing.int.testkit.ts

# Deterministic git-trackable status snapshot
npx @elench/testkit --type int --write-status

# Lifecycle
npx @elench/testkit status
npx @elench/testkit destroy
npx @elench/testkit cleanup

# Inspect the latest run artifact
npx @elench/testkit show
npx @elench/testkit show __testkit__/health/health.int.testkit.ts
npx @elench/testkit artifacts __testkit__/health/health.int.testkit.ts
npx @elench/testkit logs __testkit__/health/health.int.testkit.ts
npx @elench/testkit watch

# Known-failures tooling
npx @elench/testkit known-failures validate --issue-mode error
npx @elench/testkit known-failures render --output KNOWN_FAILURES.md

# Capture a template DB schema snapshot
npx @elench/testkit db snapshot capture --service api --output scripts/testkit/schema-baseline.sql
```

`testkit` now keeps the default terminal output intentionally short: one line
per completed file, a concise failure block, and a final summary. Service logs,
captured runtime output, emitted artifacts, and user-visible LLM responses are
persisted under `.testkit/results/` and inspected on demand with `show`,
`artifacts`, `logs`, or `watch`.

`testkit discover` also maintains a small durable per-test history index at
`.testkit/history/tests.json`. The index tracks first/last seen timestamps,
run counts, pass/fail/skip counts, average duration, and last observed status,
and those summaries are exposed in compact, verbose, and JSON discovery output.

## Setup

Create `testkit.setup.ts` at repo root:

```ts
import {
  commandStep,
  defineTestkitSetup,
  localDatabase,
  nextService,
  nodeToolchain,
  schemaSql,
  seedCommand,
  seededDatabaseTemplate,
  service,
  tsxService,
  verifyModule,
} from "@elench/testkit/setup";

export default defineTestkitSetup({
  execution: {
    workers: 8,
    fileTimeoutSeconds: 60,
  },
  reporting: {
    knownFailuresFile: "testkit.known-failures.json",
    issueValidation: {
      provider: "github",
      mode: "warn",
      cacheTtlSeconds: 900,
    },
  },
  toolchains: {
    frontendNode: nodeToolchain({
      cwd: "frontend",
      detect: "auto",
      install: "download",
    }),
  },
  services: {
    api: service({
      ...tsxService({
        cwd: ".",
        entry: "src/index.ts",
        port: 3004,
        readyPath: "/health",
      }),
      envFiles: [".env.testkit"],
      database: localDatabase({
        template: seededDatabaseTemplate({
          inputs: ["db/schema.sql", "scripts/seed.ts"],
          schema: schemaSql("db/schema.sql"),
          seed: seedCommand("npm run db:seed"),
          verify: verifyModule("src/testkit/verify-seed.ts#verifySeed"),
        }),
      }),
      runtime: {
        instances: 1,
        maxConcurrentTasks: 4,
      },
      requirements: {
        files: [
          {
            path: "src/api/routes/__testkit__/crawls-worker-concurrency.int.testkit.ts",
            locks: ["global-worker-loop"],
          },
        ],
      },
    }),
    frontend: service({
      ...nextService({
        cwd: "frontend",
        port: 3000,
        start: "./node_modules/.bin/next start --port {port}",
        env: {
          NEXT_DIST_DIR: "{prepareDir}/dist",
          NEXT_PUBLIC_API_URL: "{baseUrl:api}",
        },
      }),
      dependsOn: ["api"],
      envFiles: ["frontend/.env.testkit"],
      runtime: {
        instances: 1,
        maxConcurrentTasks: 2,
        toolchain: "frontendNode",
        prepare: {
          inputs: ["frontend/src", "frontend/public", "frontend/package.json"],
          steps: [commandStep("npm run build", { cwd: "frontend" })],
        },
      },
    }),
    billing: service({
      skip: {
        files: [
          {
            path: "__testkit__/invoices/invoices.int.testkit.ts",
            reason: "Billing is still stubbed locally",
          },
        ],
        suites: [
          {
            selector: "pw:lifecycle",
            reason: "End-to-end billing lifecycle is not implemented yet",
          },
        ],
      },
    }),
  },
});
```

`testkit.setup.ts` is optional for simple repos, but it is the primary escape hatch
for:

- worker count and per-file runtime budget
- per-file wall clock timeout budget
- multi-service graphs
- local runtime instance counts
- per-runtime concurrent task caps
- repo-managed Node toolchains for prepare/start commands
- one-time runtime preparation steps for stable shared servers
- local DB binding configuration
- template database migrate / seed / verify stages
- template schema snapshot capture
- explicit per-file or per-suite locks
- named HTTP suite profiles
- known-failure annotation merge for enriched status/run artifacts
- optional GitHub-backed known-failure issue validation
- repo-declared suite/file skip policies with explicit reasons
- telemetry upload configuration

`runtime.prepare` is the generic build-once hook for shared runtimes. It runs
once per runtime generation before local services start, fingerprints declared
inputs, and writes cache state under the service runtime directory. This is the
right way to move expensive browser targets from `next dev` / watch mode to
stable build-and-start flows.

`database.template` is the database-side equivalent for reusable template DB
state. It always executes in three explicit phases:

- `migrate`
- `seed`
- `verify`

For most repos, prefer the intent-focused helpers:

- `schemaSql(...)`
- `seedCommand(...)`
- `seedModule(...)`
- `verifyCommand(...)`
- `verifyModule(...)`
- `seededDatabaseTemplate(...)`

Use raw `commandStep(...)`, `sqlFileStep(...)`, and `moduleStep(...)` arrays when
you need lower-level control over the exact stage layout.

`runtime.toolchain` is the first-class way to make those prepare/start commands
run under the correct Node toolchain instead of whatever `node`/`npm` happened
to launch `testkit`. Node toolchains support:

- host verification mode: `install: "require-host"`
- cached repo-local provisioning mode: `install: "download"`
- auto-detection from:
  - `package.json#volta.node`
  - `.nvmrc`
  - `.node-version`
  - `.tool-versions` (`nodejs`)
  - `package.json#engines.node`
  - `package.json#volta.npm`
  - `package.json#packageManager`
  - `package.json#engines.npm`

Example:

```ts
toolchains: {
  frontendNode: nodeToolchain({
    cwd: "frontend",
    detect: "auto",
    install: "download",
  }),
},
services: {
  frontend: service({
    runtime: {
      toolchain: "frontendNode",
    },
  }),
}
```

If `reporting.knownFailuresFile` is configured, `testkit` enriches
`.testkit/results/latest.json` and `testkit.status.json` with:

- per-file `failureDetails`
- per-file `triage` metadata (issue, classification, description)
- top-level `triageSummary` counts for known vs untriaged failures

Known-failure entry authoring uses this contract:

- `title`
  - exact issue-tracker title for the linked issue
  - shared issues may reuse the same title across multiple local entries
- `description`
  - product-local bug slice or route-family summary
  - this is where file-specific nuance belongs
- `whyFailing`
  - underlying technical cause of the failure

If `reporting.issueValidation` is also configured, `testkit` validates known-failure
issue references against GitHub and adds top-level `knownFailuresIssueValidation`
data to the run/status artifacts. The most important stale-triage signal is:

- a known-failure test still fails, but the linked GitHub issue is closed

In `mode: "error"`, exact GitHub metadata drift is also treated as a validation
failure:

- title does not match the linked issue title
- local open/closed state does not match the linked issue state

Reproduction warnings are execution-aware:

- `failed` means the known failure reproduced
- `passed` means the matched test executed and did not reproduce
- `skipped` and `not_run` do not count as reproduction evidence

Known-failure operations are also available as first-class CLI commands:

- `testkit known-failures validate`
- `testkit known-failures render`

## Authoring

HTTP suites:

```ts
import { defineHttpSuite } from "@elench/testkit";

const suite = defineHttpSuite(({ rawReq }) => {
  rawReq("GET", "/health");
});

export default suite;
```

`testkit` suite files should default-export the suite object returned by
`defineHttpSuite(...)` or `defineDalSuite(...)`.

Named HTTP profiles live in `testkit.setup.ts` and can be referenced by name:

```ts
import { defineHttpSuite } from "@elench/testkit";

const suite = defineHttpSuite({ profile: "default-auth" }, ({ req, setupData }) => {
  req("GET", "/api/auth/session", setupData);
});
```

DAL suites:

```ts
import { defineDalSuite } from "@elench/testkit";

const suite = defineDalSuite(({ db }) => {
  db.query("select 1");
});

export default suite;
```

Scenario suites:

```ts
import { defineScenarioSuite } from "@elench/testkit";

const suite = defineScenarioSuite(({ rawReq, scenario }) => {
  const plan = scenario.choose("journey", {
    endpoint: scenario.pick("endpoint", ["/health", "/message"]),
    includeHealthCheck: scenario.maybe("includeHealthCheck", 1),
  });

  const selected = scenario.resource("selected-endpoint", () => rawReq("GET", plan.endpoint));

  scenario.step("fetch selected endpoint", () => {
    selected.get();
  });
});

export default suite;
```

Low-level runtime primitives remain available:

```ts
import { check, group, http, waitFor } from "@elench/testkit/runtime";
```

`waitFor()` consumes the file budget configured by `execution.fileTimeoutSeconds`.
Consumers should not set local timeout values in test files.

```ts
import { waitFor } from "@elench/testkit/runtime";

const response = waitFor(
  () => req("GET", "/api/v1/jobs/123", setupData),
  (res) => JSON.parse(res.body).data?.status === "completed",
  { description: "job 123 to complete" }
);
```

## Discovery

`testkit` discovers suites from `__testkit__/` directories.

Example layouts:

- `src/api/routes/__testkit__/auth/me.int.testkit.ts`
- `src/db/__testkit__/sessions/count-type.dal.testkit.ts`
- `frontend/__testkit__/navigation/navigation.pw.testkit.ts`
- `src/internal/handler/__testkit__/repos/crud.int.testkit.ts`

`testkit` uses these suffixes automatically:

- `*.int.testkit.ts`
- `*.e2e.testkit.ts`
- `*.scenario.testkit.ts`
- `*.dal.testkit.ts`
- `*.load.testkit.ts`
- `*.pw.testkit.ts`

Ownership is inferred from:

- the deepest matching service root from `services.<name>.local.cwd`
- optional `services.<name>.discovery.roots` overrides for shared-root edge cases

Suite names are inferred from the colocated path:

- `auth/__testkit__/*.int.testkit.ts` => `auth`
- `routes/__testkit__/auth/*.int.testkit.ts` => `auth`

Discovery is also a first-class CLI/API surface:

- `testkit discover`
  - human-first compact output with service -> type -> suite -> file hierarchy
- `testkit discover --output-mode verbose`
  - explicit paths, IDs, locks, dependencies, skip reasons, and history detail
- `testkit discover --json`
  - machine-readable output with stable enums, canonical paths, and summary data
- `testkit discover --output .testkit/discovery.json`
  - writes the same machine-readable JSON document to a file artifact

Compact mode prefers derived human labels such as `Agent Configs Auth Gate`
instead of printing long file paths as the primary row label. Exact paths remain
available in verbose and JSON output.

The public API is exported from `@elench/testkit/discovery`:

```ts
import { discoverTests } from "@elench/testkit/discovery";

const result = await discoverTests({
  dir: process.cwd(),
  runnableOnly: true,
  diagnostics: "report",
});
```

JSON and file output are machine-first. Each discovered file carries a stable
identifier plus canonical metadata such as:

- `id`
- `path`
- `service`
- `suiteName`
- `selectionType`
- `internalType`
- `framework`
- `skipped`
- `skipReason`
- `locks`
- `dependsOn`
- `displayName`
- `history`

Discovery history is generic and local to `testkit`. `firstSeenAt` is derived
from the first time a file appears in the history index, not from filesystem or
Git metadata.

## Local Databases

`@elench/testkit` provisions Docker-managed local Postgres automatically for
services that define `database: localDatabase(...)`.

- template databases are cached
- runtime databases are cloned from templates when binding is `per-runtime`
- shared databases are reused when binding is `shared`
- template fingerprints are derived automatically from env files, migrate/seed
  config, and repo contents

Manual `template.inputs` overrides are still available for edge cases.

## Development Tests

```bash
npm test
npm run test:unit
npm run test:integration
npm run test:system
```
