# @elench/testkit

`@elench/testkit` discovers `*.testkit.ts` files, infers suite ownership from the
filesystem, starts local services, provisions Docker-managed local Postgres
databases, and runs HTTP, DAL, and Playwright suites.

The package is now driven by `testkit.setup.ts`, not `testkit.config.json`.

## Usage

```bash
cd my-product

# Run every testkit-managed suite
npx @elench/testkit

# Filter by type
npx @elench/testkit --type int
npx @elench/testkit --type dal
npx @elench/testkit --type e2e
npx @elench/testkit --type int,e2e,dal
npx @elench/testkit --type pw

# Parallel file execution
npx @elench/testkit --workers 8

# One file-level wall clock budget for every suite file
npx @elench/testkit --file-timeout-seconds 60

# Run a deterministic shard
npx @elench/testkit --shard 1/3

# Specific service / suite
npx @elench/testkit --service frontend --type pw -s navigation
npx @elench/testkit --service api --type int -s health
npx @elench/testkit --type int,e2e,dal -s dal:queries

# Exact file
npx @elench/testkit --type int --file __testkit__/health/health.int.testkit.ts

# Temporarily ignore repo-declared skip rules
npx @elench/testkit --ignore-skip-rules --file __testkit__/billing/billing.int.testkit.ts

# Deterministic git-trackable status snapshot
npx @elench/testkit --type int --write-status

# Lifecycle
npx @elench/testkit status
npx @elench/testkit destroy

# Known-failures tooling
npx @elench/testkit known-failures validate --issue-mode error
npx @elench/testkit known-failures render --output KNOWN_FAILURES.md

# Capture a template DB schema snapshot
npx @elench/testkit db snapshot capture --service api --output scripts/testkit/schema-baseline.sql
```

## Setup

Create `testkit.setup.ts` at repo root:

```ts
import {
  commandStep,
  defineTestkitSetup,
  localDatabase,
  moduleStep,
  nextService,
  service,
  sqlFileStep,
  tsxService,
} from "@elench/testkit/setup";

export default defineTestkitSetup({
  execution: {
    workers: 8,
    fileTimeoutSeconds: 60,
  },
  reporting: {
    knownFailuresFile: "testkit.known-failures.json",
    issueValidation: {
      provider: "github",
      mode: "warn",
      cacheTtlSeconds: 900,
    },
  },
  services: {
    api: service({
      ...tsxService({
        cwd: ".",
        entry: "src/index.ts",
        port: 3004,
        readyPath: "/health",
      }),
      envFiles: [".env.testkit"],
      database: localDatabase({
        template: {
          inputs: ["db/schema.sql", "scripts/seed.ts"],
          migrate: [sqlFileStep("db/schema.sql")],
          seed: [commandStep("npm run db:seed")],
          verify: [moduleStep("src/testkit/verify-seed.ts#verifySeed")],
        },
      }),
      runtime: {
        instances: 1,
        maxConcurrentTasks: 4,
      },
      requirements: {
        files: [
          {
            path: "src/api/routes/__testkit__/crawls-worker-concurrency.int.testkit.ts",
            locks: ["global-worker-loop"],
          },
        ],
      },
    }),
    frontend: service({
      ...nextService({
        cwd: "frontend",
        port: 3000,
        start: "./node_modules/.bin/next start --port {port}",
        env: {
          NEXT_DIST_DIR: "{prepareDir}/dist",
          NEXT_PUBLIC_API_URL: "{baseUrl:api}",
        },
      }),
      dependsOn: ["api"],
      envFiles: ["frontend/.env.testkit"],
      runtime: {
        instances: 1,
        maxConcurrentTasks: 2,
        prepare: {
          inputs: ["frontend/src", "frontend/public", "frontend/package.json"],
          steps: [commandStep("npm run build", { cwd: "frontend" })],
        },
      },
    }),
    billing: service({
      skip: {
        files: [
          {
            path: "__testkit__/invoices/invoices.int.testkit.ts",
            reason: "Billing is still stubbed locally",
          },
        ],
        suites: [
          {
            selector: "pw:lifecycle",
            reason: "End-to-end billing lifecycle is not implemented yet",
          },
        ],
      },
    }),
  },
});
```

`testkit.setup.ts` is optional for simple repos, but it is the primary escape hatch
for:

- worker count and per-file runtime budget
- per-file wall clock timeout budget
- multi-service graphs
- local runtime instance counts
- per-runtime concurrent task caps
- one-time runtime preparation steps for stable shared servers
- local DB binding configuration
- template database migrate / seed / verify stages
- template schema snapshot capture
- explicit per-file or per-suite locks
- named HTTP suite profiles
- known-failure annotation merge for enriched status/run artifacts
- optional GitHub-backed known-failure issue validation
- repo-declared suite/file skip policies with explicit reasons
- telemetry upload configuration

`runtime.prepare` is the generic build-once hook for shared runtimes. It runs
once per runtime generation before local services start, fingerprints declared
inputs, and writes cache state under the service runtime directory. This is the
right way to move expensive browser targets from `next dev` / watch mode to
stable build-and-start flows.

If `reporting.knownFailuresFile` is configured, `testkit` enriches
`.testkit/results/latest.json` and `testkit.status.json` with:

- per-file `failureDetails`
- per-file `triage` metadata (issue, classification, description)
- top-level `triageSummary` counts for known vs untriaged failures

Known-failure entry authoring uses this contract:

- `title`
  - exact issue-tracker title for the linked issue
  - shared issues may reuse the same title across multiple local entries
- `description`
  - product-local bug slice or route-family summary
  - this is where file-specific nuance belongs
- `whyFailing`
  - underlying technical cause of the failure

If `reporting.issueValidation` is also configured, `testkit` validates known-failure
issue references against GitHub and adds top-level `knownFailuresIssueValidation`
data to the run/status artifacts. The most important stale-triage signal is:

- a known-failure test still fails, but the linked GitHub issue is closed

In `mode: "error"`, exact GitHub metadata drift is also treated as a validation
failure:

- title does not match the linked issue title
- local open/closed state does not match the linked issue state

Reproduction warnings are execution-aware:

- `failed` means the known failure reproduced
- `passed` means the matched test executed and did not reproduce
- `skipped` and `not_run` do not count as reproduction evidence

Known-failure operations are also available as first-class CLI commands:

- `testkit known-failures validate`
- `testkit known-failures render`

## Authoring

HTTP suites:

```ts
import { defineHttpSuite } from "@elench/testkit";

const suite = defineHttpSuite(({ rawReq }) => {
  rawReq("GET", "/health");
});

export default suite;
```

`testkit` suite files should default-export the suite object returned by
`defineHttpSuite(...)` or `defineDalSuite(...)`.

Named HTTP profiles live in `testkit.setup.ts` and can be referenced by name:

```ts
import { defineHttpSuite } from "@elench/testkit";

const suite = defineHttpSuite({ profile: "default-auth" }, ({ req, setupData }) => {
  req("GET", "/api/auth/session", setupData);
});
```

DAL suites:

```ts
import { defineDalSuite } from "@elench/testkit";

const suite = defineDalSuite(({ db }) => {
  db.query("select 1");
});

export default suite;
```

Low-level runtime primitives remain available:

```ts
import { check, group, http, waitFor } from "@elench/testkit/runtime";
```

`waitFor()` consumes the file budget configured by `execution.fileTimeoutSeconds`.
Consumers should not set local timeout values in test files.

```ts
import { waitFor } from "@elench/testkit/runtime";

const response = waitFor(
  () => req("GET", "/api/v1/jobs/123", setupData),
  (res) => JSON.parse(res.body).data?.status === "completed",
  { description: "job 123 to complete" }
);
```

## Discovery

`testkit` discovers suites from `__testkit__/` directories.

Example layouts:

- `src/api/routes/__testkit__/auth/me.int.testkit.ts`
- `src/db/__testkit__/sessions/count-type.dal.testkit.ts`
- `frontend/__testkit__/navigation/navigation.pw.testkit.ts`
- `src/internal/handler/__testkit__/repos/crud.int.testkit.ts`

`testkit` uses these suffixes automatically:

- `*.int.testkit.ts`
- `*.e2e.testkit.ts`
- `*.dal.testkit.ts`
- `*.load.testkit.ts`
- `*.pw.testkit.ts`

Ownership is inferred from:

- the deepest matching service root from `services.<name>.local.cwd`
- optional `services.<name>.discovery.roots` overrides for shared-root edge cases

Suite names are inferred from the colocated path:

- `auth/__testkit__/*.int.testkit.ts` => `auth`
- `routes/__testkit__/auth/*.int.testkit.ts` => `auth`

## Local Databases

`@elench/testkit` provisions Docker-managed local Postgres automatically for
services that define `database: localDatabase(...)`.

- template databases are cached
- runtime databases are cloned from templates when binding is `per-runtime`
- shared databases are reused when binding is `shared`
- template fingerprints are derived automatically from env files, migrate/seed
  config, and repo contents

Manual `template.inputs` overrides are still available for edge cases.

## Development Tests

```bash
npm test
npm run test:unit
npm run test:integration
npm run test:system
```
