# Long Tail

Write durable workflows in TypeScript. When they need a human, they escalate. When they need AI, they orchestrate. When a pattern repeats, they compile it away. Postgres is the engine.

```bash
npm install @hotmeshio/long-tail
```

## How it works

You write a workflow function. Each activity call checkpoints to Postgres — if the process crashes, it resumes from the last completed step.

```typescript
import { Durable } from '@hotmeshio/hotmesh';
import type { LTEnvelope } from '@hotmeshio/long-tail';
import * as activities from './activities';

const { analyzeContent } = Durable.workflow.proxyActivities<typeof activities>({ activities });

export async function reviewContent(envelope: LTEnvelope) {
  const analysis = await analyzeContent(envelope.data.content);

  if (analysis.confidence >= 0.85) {
    return { type: 'return' as const, data: { approved: true, analysis } };
  }

  // Low confidence — escalate to a human reviewer
  return {
    type: 'escalation' as const,
    role: 'reviewer',
    message: `Review needed (confidence: ${analysis.confidence})`,
    data: { content: envelope.data.content, analysis },
  };
}
```

That's a complete workflow. It runs, checkpoints, and when confidence is low, it hands off to a human. The human resolves it through the dashboard or API, and the workflow completes. No separate queue system, no webhook callbacks — the escalation is part of the execution.

Activities are plain functions:

```typescript
export async function analyzeContent(content: string) {
  const result = await llm.classify(content);
  return { confidence: result.confidence, flags: result.flags };
}
```

## Start

Point at Postgres. Everything else is optional.

```typescript
import { start } from '@hotmeshio/long-tail';

const lt = await start({
  database: { host: 'localhost', port: 5432, user: 'postgres', password: 'password', database: 'mydb' },
  workers: [{ taskQueue: 'default', workflow: reviewContent }],
});
```

Dashboard at [http://localhost:3000](http://localhost:3000). The [boilerplate](https://github.com/hotmeshio/long-tail-boilerplate) has a working project with workflows, MCP servers, and MinIO.

## The pattern

Most systems start with a durable workflow and stop there. Long Tail keeps going.

**Step 1 — Author a durable workflow.** Your function checkpoints to Postgres. It can sleep, branch, call child workflows, wait for signals. Standard durable execution.

**Step 2 — Certify it.** Promotion to certified adds interceptor guarantees: failures escalate instead of throwing, escalation chains route through RBAC-scoped roles, and every error is either handled or surfaced. It cannot silently fail.

```bash
curl -X PUT http://localhost:3000/api/workflows/reviewContent/config \
  -H "Authorization: Bearer $TOKEN" \
  -d '{ "invocable": true, "task_queue": "default", "default_role": "reviewer" }'
```

**Step 3 — React to events.** Workflows publish topics. Agents subscribe. When `activity.failed` fires, an automation can re-run the step, notify a team, or trigger a different workflow. The choreography is dynamic — add subscribers through the dashboard without changing code.

**Step 4 — Compile what repeats.** The Pipeline Designer takes a working execution and compiles it into a deterministic YAML DAG. No LLM at runtime, no replay overhead, typed inputs and outputs. It deploys as a reusable tool that any workflow or API call can invoke.

Over time, the system accumulates compiled tools. Problems that once required a human, then required AI reasoning, eventually require neither.

## Register MCP tools

Long Tail connects to any MCP server. Registered tools become durable activities and are available to the Pipeline Designer.

**Existing package — no code:**

```bash
curl -X POST http://localhost:3000/api/mcp/servers \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "filesystem",
    "transport_type": "stdio",
    "transport_config": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-filesystem", "/data"] },
    "tags": ["files", "storage"],
    "auto_connect": true
  }'
```

**Remote server — point at a URL:**

```bash
curl -X POST http://localhost:3000/api/mcp/servers \
  -d '{ "name": "my-python-server", "transport_type": "sse", "transport_config": { "url": "http://python-service:8000/mcp" } }'
```

**In-process — write your own:**

```typescript
import { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js';
import { z } from 'zod';
import { registerMcpTool } from '@hotmeshio/long-tail';

export function createImageToolsServer(): McpServer {
  const server = new McpServer({ name: 'image-tools', version: '1.0.0' });

  registerMcpTool(server, 'resize_image', 'Resize an image.', {
    path: z.string().describe('Path to the image'),
    width: z.number().optional(),
    height: z.number().optional(),
  }, async (args: any) => ({
    content: [{ type: 'text', text: JSON.stringify(await resize(args)) }],
  }));

  return server;
}
```

```typescript
const lt = await start({
  // ...
  mcp: { serverFactories: { 'image-tools': createImageToolsServer } },
});
```

All three paths produce the same outcome: tools callable as durable activities. See the [MCP guide](docs/mcp.md).

## Compile workflows

The `ltc` compiler scans TypeScript workflow files and compiles them to YAML DAGs — like `tsc` for workflows.

```bash
export ANTHROPIC_API_KEY=sk-ant-...
npx ltc compile workflows/
```

The source is the spec. The compiled YAML is the optimized execution. Both live in the repo. See the [Compiler Guide](docs/compiler.md).

## Full configuration

```typescript
const lt = await start({
  database: { connectionString: process.env.DATABASE_URL },
  workers: [{ taskQueue: 'default', workflow: reviewContent }],

  // Everything below is optional
  seed: { admin: { externalId: 'admin', password: process.env.ADMIN_PASSWORD } },
  mcp: { server: { enabled: true }, serverFactories: { 'my-tools': createMyToolsServer } },
  escalation: { strategy: 'mcp' },
  auth: { secret: process.env.JWT_SECRET },
  telemetry: { honeycomb: { apiKey: process.env.HNY } },
  logging: { pino: { level: 'info' } },
  maintenance: true,
});
```

## Embed in an existing app

Long Tail runs as an embedded package inside NestJS, Express, or any Node.js application. No extra HTTP server, no extra ports.

```typescript
import { start, createClient } from '@hotmeshio/long-tail';

const lt = await start({
  database: { connectionString: process.env.DATABASE_URL },
  server: { enabled: false },
  seed: { admin: { externalId: 'system' } },
  workers: [{ taskQueue: 'default', workflow: reviewContent }],
});

const client = createClient({ auth: { userId: lt.adminUserId } });

const tasks = await client.tasks.list({ status: 'completed', limit: 10 });
const result = await client.escalations.claim({ id: 'esc_123', durationMinutes: 30 });
```

Mount the dashboard at a subpath:

```typescript
import { LTExpressAdapter } from '@hotmeshio/long-tail';

const adapter = new LTExpressAdapter();
adapter.setBasePath('/admin/longtail');
app.use('/admin/longtail', adapter.getRouter());
```

Subscribe to events with callbacks:

```typescript
client.events.on('task.completed', (event) => console.log('done:', event.workflowId));
client.events.on('escalation.*', (event) => notifyTeam(event));
```

Every SDK call returns an `LTApiResult` — same status codes, same validation, same RBAC. See the [SDK guide](docs/sdk.md).

## Deployment

Three modes from the same codebase:

```typescript
// Standalone — dashboard + API + workers
await start({ database: { connectionString: process.env.DATABASE_URL } });

// Worker-only — no HTTP server
await start({ database: { connectionString: process.env.DATABASE_URL }, server: { enabled: false }, workers: [...] });

// Embedded — inside your app, SDK calls only
await start({ database: { connectionString: process.env.DATABASE_URL }, server: { enabled: false } });
const lt = createClient({ auth: { userId: 'service' } });
```

All modes share PostgreSQL and scale independently. See [Cloud Deployment](docs/cloud.md).

## Docs

| Guide | What it covers |
|-------|---------------|
| [The Long Tail Story](docs/story.md) | Why this exists, what accumulates over time |
| [Workflows](docs/workflows.md) | Activities, interceptor, escalation lifecycle, composition |
| [IAM](docs/iam.md) | Identity propagation, service accounts, credential exchange |
| [Dashboard](docs/dashboard.md) | Navigation, key pages, event feed |
| [MCP](docs/mcp.md) | Server registration, tool calls, human queue |
| [Compilation](docs/compilation.md) | Dynamic to deterministic pipeline wizard |
| [Compiler](docs/compiler.md) | `ltc compile` — durable TypeScript to YAML DAGs |
| [CLI](docs/cli.md) | `ltc` — terminal access to workflows, escalations, knowledge, MCP |
| [Escalation Strategies](docs/escalation-strategies.md) | Default, MCP triage, custom handlers |
| [SDK](docs/sdk.md) | Embedded usage, `createClient`, event subscriptions |
| [Architecture](docs/architecture.md) | Project structure, conventions, discovery |
| [Cloud](docs/cloud.md) | AWS ECS, GCP Cloud Run, Docker |
| [Data Model](docs/data.md) | Database schema |

**Adapters:** [Auth](docs/auth.md) · [Events](docs/events.md) · [Telemetry](docs/telemetry.md) · [Logging](docs/logging.md) · [Maintenance](docs/maintenance.md) · [OAuth](docs/oauth-and-delegation.md)

**HTTP API:** [Workflows](docs/api/http/workflows.md) · [Tasks](docs/api/http/tasks.md) · [Escalations](docs/api/http/escalations.md) · [YAML Workflows](docs/api/http/yaml-workflows.md) · [Users](docs/api/http/users.md) · [Roles](docs/api/http/roles.md) · [Service Accounts](docs/api/http/service-accounts.md) · [MCP Servers](docs/api/http/mcp-servers.md) · [Pipelines](docs/api/http/pipelines.md) · [Exports](docs/api/http/exports.md)

**SDK:** [Overview](docs/sdk.md) · [Workflows](docs/api/sdk/workflows.md) · [Tasks](docs/api/sdk/tasks.md) · [Escalations](docs/api/sdk/escalations.md) · [YAML Workflows](docs/api/sdk/yaml-workflows.md) · [MCP](docs/api/sdk/mcp.md) · [Events](docs/api/sdk/events.md)

## Contributing

```bash
git clone https://github.com/hotmeshio/long-tail.git
cd long-tail
docker compose up -d --build
```

Open [http://localhost:3000](http://localhost:3000). Example workflows seed the dashboard.

| User | Password | Role |
|------|----------|------|
| `superadmin` | `l0ngt@1l` | superadmin |
| `admin` | `l0ngt@1l` | admin |
| `engineer` | `l0ngt@1l` | engineer |
| `reviewer` | `l0ngt@1l` | reviewer |

See [Contributing](docs/contributing.md).

## License

See [LICENSE](LICENSE).
