# Design: LangWatch Trace Experience v2

Generated by /office-hours on 2026-04-22
Branch: main
Repo: observe-exp
Status: APPROVED
Mode: Startup

## Problem Statement

LangWatch's trace viewer is a 1.5-2 year old MVP built for a world where AI meant "prompt optimization." The world has shifted to autonomous agents with complex tool call chains, multi-step reasoning, and long-running tasks. The current product is too simple, too slow, poorly designed, and built around the wrong abstraction. Users don't think in "traces" — they think in conversations, sessions, errors, cost, and system health. The product forces everyone through a one-size-fits-all trace lens regardless of their role or intent.

The rebuild scope: new frontend + new API layer, both behind a feature flag. ClickHouse and ingestion engine stay. Old and new coexist until the new experience is ready.

## Demand Evidence

- Users have directly asked for: conversation views, session grouping, error filtering, anomaly detection, live tail. None are supported.
- Internal dogfooding confirms the design is "clunky" and "looks shit" compared to competitors.
- Data density is wrong for every persona — engineers see too little technical detail, PMs see too much.
- Filtering is limited in both capability (few filterable dimensions) and usability (hard to use).
- The product isn't well-integrated with the rest of the LangWatch platform, which has matured around it.

## Status Quo

Seven competitors analyzed (Braintrust, Datadog, Grafana, Laminar, Langfuse, LangSmith, Maxim). All have converged on the same product: three-pane IDE layout, hierarchical tree/waterfall, LLM metrics (tokens, cost, latency, model), "Open in Playground" button. The market has commoditized the trace viewer.

Universal gaps across all competitors:
- No real-time collaboration features (cursor presence, comments, tagging) across any vendor. Braintrust has shared/saved views but no multiplayer.
- No non-engineer access patterns (every vendor assumes developer)
- No intelligent trace summarization
- No critical path analysis
- No semantic search over trace content
- No cost anomaly attribution
- No simulation/expected-vs-actual comparison

## Target User & Personas

The core insight: everything is trace data underneath, but six+ different people walk into the same room of data needing completely different views.

| Persona | What they want to see | The "lens" |
|---|---|---|
| Product owner | Conversations, frustration, user sentiment | Conversation lens |
| Engineer | System performance, service health, spans | Engineering lens |
| Data scientist | Prompt performance across all interactions | Prompt/model lens |
| QA | Real customer messages for quality checks | Message lens |
| Compliance | Messages filtered for policy violations | Compliance lens |
| Finance | Cost by conversation, by operation, by model | Cost lens |
| Senior engineer | LLM performance, latency, token efficiency | Performance lens |

The trace is the atom. Conversations, evaluations, simulations, cost reports, compliance audits are all lenses on that atom. The product is not a trace viewer. It is a trace data engine with pluggable lenses.

Additionally, traces themselves have different types: real user conversations, simulated agent tests, evaluations, generic system traces (non-AI). The type is metadata on the trace, and each lens can filter by it.

## Constraints

- ClickHouse schema and ingestion pipeline stay as-is
- New API layer replaces the existing backend (which is also being retired)
- Feature-flagged: old and new coexist, no forced migration
- Must integrate with existing LangWatch platform (prompt management, simulations, evaluations)
- Must support OpenTelemetry trace/span data model
- Accessibility: keyboard navigation, screen reader support, WCAG 2.1 AA color contrast (especially important for compliance/enterprise personas)

## Premises

1. **Trace is the right storage model.** The UI surfaces it through multiple lenses (conversations, sessions, users, errors, services) rather than forcing a trace-first view. Traces are infrastructure; the lens is the product.
2. **The three-pane IDE layout is table stakes.** Every competitor has it. It won't win customers. Differentiation comes from what's inside it: intelligence, multiplayer, persona-awareness.
3. **New frontend + new API layer, both behind a feature flag.** ClickHouse and ingestion untouched. Old and new coexist.
4. **The moat is three things layered:** (a) a flexible engine that enables rapid experimentation with intelligence features, (b) simulation integration (unique — nobody else has this), (c) persona-aware design that's intuitive per role.
5. **Intelligence is the long-term play, but the foundation must be built for it first.** The "killer feature" isn't known yet. The architecture must support rapid experimentation.
6. **Individual records at scale aren't useful on their own.** Users with 5+ conversations/second don't want to scroll through a list. Individual records need aggregate context and intelligent surfacing to be actionable. Aggregate views and trends are a separate product layer that sits above the individual-record level.

## Cross-Model Perspective

Independent cold read challenged Premise #4: "flexible engine" is architecture, not a moat. The actual moat candidate is simulations — unique to LangWatch, hard to copy, creates a closed loop between "what happened" and "what should have happened." The team should elevate simulations from a bullet point to the centerpiece.

Also flagged: tension between "trace is the right data model" and "users don't want traces." Resolution: traces are the storage model, conversations/sessions/users are the product model. If you rebuild around traces again without nailing the grouping/linking/summarization layer, you ship a prettier version of what everyone else has.

Prototype suggestion: lead with conversations (group by session), not traces. Three hardcoded persona presets (Engineer/Product/Cost) that swap columns on the same table. Use the prototype to validate whether leading with higher-order objects feels meaningfully better.

## Approaches Considered

### Approach A: Lens Engine First
Build the dynamic lens/view engine as the core primitive. A lens defines: columns, grouping, default filters, sort order. Ship 3-4 hardcoded lenses then open it up for custom lenses.
- Effort: L
- Risk: Medium (over-engineering before knowing what lenses people want)
- Pros: Maximum flexibility, persona problem solved architecturally
- Cons: Slower to first visible value, risk of generic tool nobody loves

### Approach B: Conversation-First, Expand Out
Build the best conversation/session viewer on the market first. Group traces by session, render as chat transcripts, show frustration signals, cost per conversation. Add other views later.
- Effort: M
- Risk: Low
- Pros: Fast to first "wow" moment, validates thesis
- Cons: Breaks down at scale (5+ conversations/second — nobody scrolls through that). Individual conversations aren't useful without aggregation. Other personas wait.

### Approach C: Trace Drawer + Multiple Entry Points
Build the trace detail drawer first. Make it world-class with multiple viz modes. Then build lightweight list views that all open this same drawer.
- Effort: M-L
- Risk: Low-Medium
- Pros: Drawer is the constant across all views, reusable
- Cons: Lists feel thin initially

### Approach D: Foundation + Intelligence Spikes (CHOSEN)
Hybrid of A and C with experimental intelligence features built in parallel.

## Recommended Approach

**Approach D: Foundation + Intelligence Spikes**

The foundation gives you something to put behind the feature flag immediately. The intelligence spikes give you the story — the thing that makes people say "this is different from every other trace viewer."

### Phase 1: Onboarding + Trace Drawer
- Empty state / onboarding view: what does this look like when you have no data? Educate users on what the product does and what they'll see when data flows in. This view adapts based on what the user wants from the product.
- Trace detail drawer: three visualization modes (Waterfall, Flame Graph, Span List — see ADR-001). Tab model for span selection (Trace Summary tab + ephemeral span tab). Rich metadata panel. This is the constant — every persona, every entry point opens this same drawer. Make it beautiful.
- Acceptance criteria for these surfaces live in `specs/traces-v2/` at the repo root.

### Definitions

A **lens** (internal term) is a named config object containing: columns, grouping, default filters, sort order, conditional formatting rules, and visualization mode. A **view** (user-facing term) is a lens rendered as a tab in the UI. A **built-in view** is a hardcoded lens shipped with the product. **Custom views** are user-created and saved. The full LensConfig schema lives in `langwatch/src/features/traces-v2/lens/schema.ts`.

### Phase 0: Schema Validation (COMPLETED)

Queried the dev ClickHouse instance. Results:

**Data volume (dev):** 70,721 trace summaries, 177,212 spans, 12,073 simulation runs, 18,511 evaluation runs.

**Grouping capabilities — all supported:**
- **By session/thread:** `ThreadId` in `analytics_trace_facts` (low usage in dev: 3 distinct threads, 6 traces). `gen_ai.conversation.id` in span attributes is richer: 19,052 spans across 1,908 distinct conversations.
- **By user:** `UserId` in `analytics_trace_facts` + `langwatch.user.id` in span attributes (14,830 spans, 20 distinct users in dev).
- **By service:** `ServiceName` in `stored_spans` — 60 distinct services (including integration tests, parity checks, real apps like `mastra-app`, `openai-agents-app`, `chainlit`).
- **By model:** `ModelNames` array in `analytics_trace_facts`, `gen_ai.request.model` in span attributes. Models seen: `openai/gpt-4o`, `openai/gpt-5-mini`, `anthropic/claude-sonnet-4-6`.
- **By prompt template:** Not a first-class column. Would need to extract from span attributes or add as metadata.

**Filterable dimensions with cardinality:**
- `ServiceName`: 60 values (LowCardinality)
- `SpanName`: 5,632 values (LowCardinality)
- `langwatch.span.type`: 15 values (llm: 63K, tool: 19K, span: 18K, evaluation: 15K, chain: 13K, agent: 12K, module: 11K, rag: 6K, etc.)
- `Labels`: array field, e.g. "production", "metadata-check"
- `TopicId`/`SubTopicId`: available in both tables (11,296 traces with topic in summaries)
- `ContainsError`: boolean flag
- `SatisfactionScore`: nullable float (11,296 traces have satisfaction scores)
- `Models`: array of model names
- `EventTypes`, `EventScoreKeys`: array fields for evaluation results
- `ThumbsUpDownVote`: nullable int8

**Trace complexity:** avg 3.26 spans/trace, max 299 spans/trace in dev. Production will be higher. The 500ms SLA for 1000-span traces needs testing against production data.

**Simulation data is rich:** `simulation_runs` has `Messages.Role`, `Messages.Content`, `Messages.TraceId` arrays + `Verdict`, `Reasoning`, `MetCriteria`, `UnmetCriteria`. Direct link between simulations and traces via `TraceIds` array. Simulation diff (Phase 4) is feasible with the current schema.

**Verdict: Phase 2 is fully buildable on the current schema.** Session grouping uses `ThreadId` or `gen_ai.conversation.id`. Service grouping uses `ServiceName`. Model grouping uses `ModelNames`. Error filtering uses `ContainsError`. Cost filtering uses `TotalCost`. The only gap is prompt template as a first-class grouping dimension.

### Phase 2: Lens Engine (mock data, this repo)
- User-configurable views: save, edit, delete, rename custom views
- Column configuration: show/hide, drag-to-reorder, resize
- Grouping engine: group traces by session, service, user, or model (accordion groups)
- 3 additional built-in views: "By Model", "By Service", "By User"
- Conditional formatting: color-code cells by value thresholds
- View analytics: invisible instrumentation tracking view usage patterns
- Acceptance criteria for each component live in `specs/traces-v2/` at the repo root.

### Phase 3A: App Networking Layer (production repo)
- State management: TanStack Query (server state via @trpc/tanstack-query) + Zustand with 4 slices (filter, view, drawer, UI)
- Progressive data loading: 5-level model (table -> drawer header -> span skeleton -> span detail -> accordion detail)
- Filter state machine: AST as source of truth, two-way sync between search bar and facets
- Mock data hooks replaced with real tRPC calls. Components stay untouched.

### Phase 3B: tRPC Backend (production repo)
- New API layer connecting the frontend to ClickHouse
- Replaces the existing backend
- Design to be done when Phase 3A is underway

### Phase 4+: Intelligence Features (production repo, future)
- **"What went wrong?" button:** One click on any trace. AI walks the execution tree, finds the decision point where things diverged from expected, returns a 3-sentence diagnosis. Not a summary, a diagnosis.
- **AI-generated views:** The UI includes a text box: "What are you investigating?" AI generates a LensConfig (columns, filters, grouping, visualization mode). The lens engine renders it. Users never configure anything manually.
- **Frustration detection:** Sentiment analysis on conversation traces. Detect patterns like repeated questions, "that's not what I asked," escalation language. Similar to Datadog's frustrated clicks, but for AI conversations.
- **Simulation diff:** Side-by-side comparison: "what happened in prod" vs "what happened in simulation." LangWatch's unique competitive asset. Nobody else can show the delta between real behavior and expected behavior.

**Fallback behavior:** When AI diagnosis times out or returns low-confidence results, show "Diagnosis unavailable, view raw trace" with a direct link to the trace drawer. When AI-generated views produce a bad config, show the default "All Traces" view with a message "Couldn't generate a view for that query, try rephrasing or use a view." Include a thumbs-up/down feedback mechanism on all AI outputs to measure quality and build training signal.

### Future: Multiplayer (separate scope, post-v2 launch)
- Cursor presence (see which traces teammates are looking at), comments, tagging (notifications to Slack)
- Requires separate real-time infrastructure (WebSocket/Liveblocks layer outside ClickHouse)
- This is effectively a second product layer and should be scoped independently

### Future: Aggregate Intelligence
- Trend views above the individual-record level
- Cost anomaly attribution
- Semantic search over trace content
- Critical path analysis
- Agent/service maps showing bottlenecks
- This is the "intelligence" layer that requires the foundation to be in place first

## Product Philosophy

**Observability that thinks for you.** Not a dashboard you configure. Not a trace viewer you read. A system that understands your agents, answers your questions, and shows you the proof.

Three pillars:
1. **Ask, don't configure.** Tell the product what you're investigating. It assembles the right view.
2. **Diagnose, don't display.** One click: here's what went wrong, here's the decision point, here's the proof.
3. **Compare, don't guess.** Show the delta between real and expected. Simulations aren't a separate product — they're the other half of observability.

## Open Questions

1. **What does the AI-generated view architecture look like?** Recommended: the LLM generates a lens config object (see Definitions), not SQL or React components. This constrains the output to a known schema, makes validation trivial, and keeps the LLM out of the rendering path. This must be resolved before Phase 3 work begins.
2. **How do you handle trace types in the UI?** A simulation trace, an eval trace, and a real user trace are all "traces" in ClickHouse, but mean very different things. How explicit is the type in the UI? Is it a filter? A lens preset? A badge?
3. **What does the "What went wrong?" diagnosis actually need?** Does it need access to the prompt/model context? The simulation baseline? The evaluation criteria? Or is the trace alone enough for most diagnoses?
4. **Where does multiplayer state live?** Cursor presence, comments, and tags are real-time collaborative features. This likely needs a separate real-time layer (WebSocket, Liveblocks, etc.) outside ClickHouse.
5. **How does the new API layer relate to the rest of the LangWatch API?** Is it a separate service? A new set of endpoints in the existing API? Does it share auth, permissions, team context?
6. **What's the aggregate/trend layer?** This was explicitly called "a whole other product" — but it's also where the intelligence vision lands. When does this get scoped?

## Success Criteria

- Users behind the feature flag spend more time in the new trace experience than the old one
- At least 2 personas (e.g., engineer + PM) find value in different preset views
- "What went wrong?" diagnosis is accurate enough that users trust it (>70% of diagnoses are actionable)
- Design quality matches or exceeds the best competitor (Maxim/LangSmith level polish)
- Trace drawer load time < 500ms for traces with up to 1000 spans (may require caching layer between API and ClickHouse — see Phase 0 validation)
- AI-generated views produce useful table configs >80% of the time (measured by: user does not modify the generated config before using it, tracked via instrumentation)

## Distribution Plan

Feature-flagged within the existing LangWatch web application. No separate distribution needed. Rollout plan: internal dogfooding → select design-partner customers → percentage rollout → full migration → deprecate old trace UI.

## Dependencies

- ClickHouse schema must support the grouping/linking queries needed for session, user, and service views (may need new materialized views or projections, or a caching layer for single-trace lookups)
- New API layer design and implementation
- Simulations product API for Phase 4 diff integration
- Design system / component library for consistent UI quality
- LLM integration for intelligence features (Phases 3-4)

## The Assignment

Before writing any code: take the ClickHouse MCP connection you have and run 10 queries that answer the question "what higher-order objects can we derive from the trace data we already have?" Specifically: can you group by session? By user? By service? By prompt template? What dimensions are available for filtering? What's the cardinality? The answer to these queries determines whether the lens engine is trivially buildable or requires schema work. Start there.

## What I noticed about how you think

- You said "we don't necessarily need to build what everyone else has but we need to build what everyone's going to need." That's not a safe statement. That's a bet on having better taste than the market. You then backed it up by articulating specific gaps nobody is filling.
- When I pushed for the narrowest wedge, you said "onboarding empty state." Most people building a v2 start with the power-user view. You started with the moment someone decides whether they understand the product. That's product thinking, not engineering thinking.
- You pushed back on the conversation-first approach with a real insight: "people don't really want to come in and look at individual conversations because they happen so many." You killed your own idea with a scale argument. That's intellectual honesty.
- You threw out "the UI is just a text box" as a crazy idea, but it's actually the logical endpoint of everything you described. The fact that your instinct goes there — toward making the interface disappear — says something about how you think about product.
