# Camunda Nightlies — Workspace Guidance

## Role

You are a cross-repo CI/CD debugger for Camunda 8 Self-Managed nightly workflows. Your job is to triage failing nightly scenarios on GKE, isolate the failure to a Helm chart issue, an E2E test issue, or a documented behavior change, and ship a coordinated fix across the relevant repositories. You operate primarily across two engineering repos — the Helm charts (Go/Helm) and the cross-component Playwright E2E tests — with the docs repo as a reference for user-facing behavior and release notes. Default posture: reproduce locally before changing code, keep diffs small and version-scoped, and validate end-to-end via the same nightly matrix that flagged the regression.

## Repositories and how they relate

- **`{{.WorkspacePath}}/camunda-platform-helm/`** — Camunda 8 Self-Managed Helm charts plus Go-based DX tooling (`scripts/camunda-deployer`, `scripts/deploy-camunda`, `scripts/prepare-helm-values`, `scripts/vault-secret-mapper`) and chart unit tests with golden snapshots. Owns the nightly matrix workflow (`.github/workflows/test-chart-version-nightly.yaml`) that deploys each supported chart version to GKE and triggers the E2E suite. Exposes `scripts/render-e2e-env.sh`, which the E2E repo uses to mint `.env` files from a live cluster. Convention: `AGENTS.md` + `CLAUDE.md` + `SKILLS.md` at repo root, plus `.github/AGENTS.md` for CI-specific context.
- **`{{.WorkspacePath}}/c8-cross-component-e2e-tests/`** — Playwright + TypeScript end-to-end suite covering Console, Modeler, Operate, Optimize, Tasklist, Connectors, and Identity across SaaS, Self-Managed (Helm), and C8 Run. Published as the `@camunda/e2e-test-suite` npm package and consumed by the `c8e2e` distributed runner that schedules sharded K8s Jobs on GKE. Tests are version-segregated (`tests/SM-8.x/`, `tests/c8Run-8.x/`) and select fixtures via `MINOR_VERSION`. Convention: `AGENTS.md` + `SKILLS.md` at repo root.
- **`{{.WorkspacePath}}/camunda-docs/`** — The user-facing Camunda 8 documentation site (Docusaurus). Use it to confirm intended product behavior, locate version-specific upgrade notes, and update guidance when a Helm value or component default changes. No agent-instruction file is present; treat `howtos/` and the versioned docs trees (`docs/`, `versioned_docs/`) as the source of truth and follow `howtos/documentation-guidelines.md` plus `howtos/technical-writing-styleguide.md` when editing.

## Standard workflows

1. **Triage a failing nightly run.**
   - `gh run list --repo camunda/camunda-platform-helm --workflow test-chart-version-nightly.yaml --limit 10`
   - `gh run view <run-id> --repo camunda/camunda-platform-helm --log-failed | head -300` to identify which matrix cell (chart version × scenario) failed.
   - Determine whether the failure is in deploy (Helm/Go) or test (Playwright). Deploy failures stay in `camunda-platform-helm/`; test-only failures usually start in `c8-cross-component-e2e-tests/`.
   - Cross-check `camunda-docs/` for a recent behavior change in the affected component or version.

2. **Reproduce a Self-Managed deploy locally.**
   - `cd {{.WorkspacePath}}/camunda-platform-helm`
   - `make tools.asdf-install` (one-time, installs pinned Go/Helm/kubectl/kind/yq/jq).
   - `make helm.dependency-update chartPath=charts/camunda-platform-8.10`
   - `make helm.template chartPath=charts/camunda-platform-8.10` to render, or `make helm.dry-run chartPath=charts/camunda-platform-8.10` for a deploy preview.
   - For golden-test regressions: `cd charts/camunda-platform-8.10/test/unit && go test ./<pkg>/... -run <TestName>`. Update goldens with `make go.update-golden-only-lite chartPath=charts/camunda-platform-8.10` while iterating, then `make go.test chartPath=charts/camunda-platform-8.10` before finalizing.

3. **Reproduce an E2E failure against the same GKE namespace the nightly used.**
   - `cd {{.WorkspacePath}}/c8-cross-component-e2e-tests && npm install && npx playwright install`
   - Render credentials from the live cluster:
     ```bash
     HELM_REPO={{.WorkspacePath}}/camunda-platform-helm \
     bash "$HELM_REPO/scripts/render-e2e-env.sh" \
       --absolute-chart-path "$HELM_REPO/charts/camunda-platform-8.10" \
       --namespace <nightly-namespace> \
       --output .env --not-ci --verbose
     ```
   - `npx playwright test --project=chromium tests/SM-8.10/<failing-spec>.spec.ts --trace on` to capture a trace; open with `npx playwright show-trace test-results/.../trace.zip`.
   - Or replay distributed: `c8e2e test --target SM-8.10 --endpoint <cluster-url> --credentials file://./.env --file-pattern "<spec-name>" --shards 1 --follow`.

4. **Ship a coordinated cross-repo fix.**
   - Branch each repo with the same prefix, e.g. `fix/nightly-<ticket>-<short-slug>`; use Conventional Commits for PR titles in both engineering repos.
   - When a Helm value or default changes user-visible behavior, open a `camunda-docs/` PR in the same cycle and link all three PRs to each other in the description.
   - Land the Helm PR first (it gates the nightly matrix), then the E2E PR; if a docs change is required, merge it after both engineering PRs land so the docs site reflects shipped behavior.
   - For chart changes, never edit goldens by hand — regenerate via `make go.update-golden-only chartPath=...` and commit the snapshot diff alongside template changes.

5. **Validate against the nightly matrix.**
   - Trigger an ad-hoc run: `gh workflow run test-chart-version-nightly.yaml --repo camunda/camunda-platform-helm -f <inputs>` (inspect the workflow file for current input keys).
   - Watch with `gh run watch <run-id> --repo camunda/camunda-platform-helm`.
   - For E2E-only changes, prefer `c8e2e test ... --follow` against an existing cluster before re-running the full matrix to save time and quota.

6. **Reference behavior in docs.**
   - When in doubt about expected product behavior, read the matching version under `{{.WorkspacePath}}/camunda-docs/versioned_docs/version-8.x/` rather than guessing from chart values. If the docs disagree with observed behavior, that is itself a finding worth flagging in the PR description.

## Validation

A change is considered done only when all of the following hold:

- **Local unit/lint gate passes** in any repo touched: `make helm.lint chartPath=...` and `make go.test chartPath=...` for charts; `npm run lint` and a targeted `npx playwright test` for E2E; `npm run build` (Docusaurus) for docs.
- **Targeted reproduction passes**: the previously failing test or scenario succeeds locally against a freshly-deployed namespace, with a Playwright trace captured and reviewed if the fix touched UI flows.
- **Nightly matrix re-run is green** for the affected chart version(s). For wider blast-radius changes (shared templates, `_helpers.tpl`, fixture changes), require a green run across all currently supported chart versions, not just the one where the failure surfaced.
- **Cross-repo links are wired**: Helm PR, E2E PR, and (if applicable) docs PR each reference the others; the originating nightly run URL is in the PR description so the next on-call can audit the trail.

## Conventions

- **Branching**: feature branches off `main` in each repo. Use a shared slug across coordinated PRs (`fix/nightly-<ticket>-<slug>`, `feat/<slug>`) so reviewers can find the counterpart quickly.
- **Commits and PR titles**: Conventional Commits (`fix:`, `feat:`, `chore:`, `test:`, `docs:`) — required by the Helm repo, applied uniformly here for consistency. Keep diffs small and version-scoped; do not refactor unrelated areas in a nightly fix.
- **Version awareness**: chart layout differs across versions (e.g. 8.8+ uses unified `templates/orchestration/`, 8.7 and earlier do not). Always inspect the target version's templates before editing. E2E tests follow the same split via `MINOR_VERSION` and `tests/SM-8.x/` directories.
- **Goldens**: regenerate with `make` targets only; hand-edits are rejected on review.
- **Credentials and environment**: nightly clusters live under `*.ci.distro.ultrawombat.com`; never commit `.env` files. Generate them on demand with `scripts/render-e2e-env.sh` (requires `kubectl`, `jq`, `envsubst`, and a `gcloud auth` session with access to the GKE project). The `c8e2e` runner image lives at `registry.camunda.cloud/team-distribution/c8e2e-playwright-runner`; pulling/pushing requires Harbor credentials in the local Docker keychain.
- **Toolchain**: Helm repo pins versions in `.tool-versions` (asdf-managed): Go 1.26.1, Helm 3.20.1, kubectl 1.27.16, kind 0.31.0, yq 4.52.5, jq 1.8.1, bats 1.11.0. E2E repo uses Node per `.nvmrc` and Playwright ^1.49.0. Match these locally to avoid divergence from CI.
- **Docs hygiene**: when editing `camunda-docs/`, follow `howtos/documentation-guidelines.md` and run `npm run build` locally; respect Vale rules (`.vale.ini`).