Skip to content

# Configuration

pgflow’s configuration is divided into two distinct parts, each serving a different purpose in the workflow lifecycle.

pgflow separates configuration into task execution settings (how individual steps are executed and retried) and worker settings (how the worker processes the entire flow). Task settings are stored in your database and control retry behavior, delays, and timeouts, while worker settings control concurrency, connections, and batching.

Documentation Structure

This page covers both pgflow configuration and Edge Worker configuration in one place. Future versions may separate these concerns for clarity.

Step Execution Options Configure retry behavior, timeouts, and error handling for your workflow steps. Set defaults at flow level, override per step.
Retry Behavior Understand exponential backoff and how retry delays are calculated based on your configuration.
Worker Configuration Runtime settings for EdgeWorker including concurrency, database connections, and polling behavior.
Complete Examples Real-world configuration patterns for API integrations, local processing, and mixed workloads.

Zero Configuration

pgflow comes with sensible defaults for all configuration options.

You only need to specify slugs for your flow and steps:

    new Flow({ slug: 'my_flow' })  .step({ slug: 'step1' }, handler1)  .step({ slug: 'step2' }, handler2)

## Step Execution Options

Section titled “Step Execution Options”

These settings are defined in your TypeScript flow code and compiled into SQL migrations. They control how individual steps are executed, delayed, and retried. Set defaults at the flow level, override for specific steps. Step-level options are `null` by default, inheriting from flow-level settings.

Tip

After deployment, you can update these settings without recompiling your flow. See Update Flow Options for details.

### Default Configuration

Section titled “Default Configuration”

    new Flow({  slug: 'my_flow',  maxAttempts: 3,    // max retry attempts before marking as failed  baseDelay: 5,      // initial retry delay in seconds  timeout: 60        // visibility timeout in seconds  // Note: startDelay is step-level only, not available as a default at flow level})

### `maxAttempts`

Section titled “maxAttempts”

Type: `number`
Default: `3`

The maximum number of times a task will be attempted before being marked as permanently failed.

    // Flow levelnew Flow({ slug: 'my_flow', maxAttempts: 5 })
    // Step level (overrides flow default).step({ slug: 'my_step', maxAttempts: 7 }, handler)

### `baseDelay`

Section titled “baseDelay”

Type: `number`
Default: `5`

The initial delay (in seconds) before the first retry. pgflow uses exponential backoff, so subsequent retries will have increasingly longer delays.

    // Flow levelnew Flow({ slug: 'my_flow', baseDelay: 2 })
    // Step level (overrides flow default).step({ slug: 'my_step', baseDelay: 10 }, handler)

### `timeout`

Section titled “timeout”

Type: `number`
Default: `60`

The visibility timeout (in seconds) - how long a task remains invisible to other workers while being processed.

Timeout and Task Processing

Set `timeout` higher than your task’s maximum processing time.

Here’s why:

- When a worker picks up a task, it becomes invisible for `timeout` seconds
- If processing takes longer than `timeout`, the task becomes visible again
- Other workers can then pick up and process the same task
- This leads to duplicate processing
- For example: with `timeout: 30` and a task that takes 45 seconds, the task could be processed twice

Currently, pgflow uses timeout only for visibility. In the future, the Edge Worker will also use it to terminate tasks that exceed their timeout.

    // Flow levelnew Flow({ slug: 'my_flow', timeout: 120 })
    // Step level (overrides flow default).step({ slug: 'my_step', timeout: 300 }, handler)

### `startDelay`

Section titled “startDelay”

Type: `number`
Default: `0`

Initial delay (in seconds) before task execution.

Step-level only

Unlike other options, `startDelay` cannot be set at the flow level.

Why no flow-level default?

Flow-level `startDelay` would create confusing cascading delays in DAG execution:

    With flow-level startDelay: 10s
    Time 0:   Flow startsTime 10:  Step A starts (waits 10s)Time 15:  Step A completesTime 25:  Step B starts (waits 10s after A completes)Time 30:  Step B completesTime 40:  Step C starts (waits 10s after B completes)

This results in 40+ seconds of delays, not the expected 10s.

Better alternatives:

- Need uniform delays? Use a constant as shown below
- Rate limiting? Use worker’s `maxConcurrent` setting
- Debug delays? Add only to specific steps you’re debugging
- Compliance delays? Make them explicit on relevant steps

To apply the same delay to multiple steps, use a constant:

    const RATE_LIMIT_DELAY = 2;flow  .step({ slug: "api_call_1", startDelay: RATE_LIMIT_DELAY }, handler1)  .step({ slug: "api_call_2", startDelay: RATE_LIMIT_DELAY }, handler2)

### Configuration Examples

Section titled “Configuration Examples”

#### Flow with Defaults Only

Section titled “Flow with Defaults Only”

When all steps can use the same configuration:

    new Flow({  slug: 'my_flow',  maxAttempts: 3,    // Default for all steps  baseDelay: 5,      // Default for all steps  timeout: 60        // Default for all steps})  .step({ slug: 'step1' }, handler1)  // Uses flow defaults  .step({ slug: 'step2' }, handler2)  // Uses flow defaults

#### Mixed Configuration

Section titled “Mixed Configuration”

Override flow defaults for specific steps that need different behavior:

    new Flow({  slug: 'analyze_data',  maxAttempts: 3,    // Flow defaults  baseDelay: 5,  timeout: 60})  .step({    slug: 'fetch_data',    // Uses all flow defaults  }, fetchHandler)  .step({    slug: 'process_data',    maxAttempts: 5,    // Override: more retries    timeout: 300       // Override: needs more time    // baseDelay uses flow default (5)  }, processHandler)  .step({    slug: 'call_api',    baseDelay: 10,     // Override: longer initial delay    // maxAttempts and timeout use flow defaults  }, apiHandler)

## Retry Behavior

Section titled “Retry Behavior”

pgflow uses exponential backoff for retries. The delay between attempts is calculated as:

    delay = baseDelay * 2^attemptCount

Note

Unlike the Edge Worker’s queue-only mode which supports a `maxDelay` cap, pgflow’s retry delays are not capped yet. Delays will continue to double with each attempt, at most `maxAttempts`-times, after which the step and flow are failed permanently

### Retry Delay Examples

Section titled “Retry Delay Examples”

Here’s how retry delays grow with different base delays:

Attempt | Delay (baseDelay: 2s) | Delay (baseDelay: 5s) | Delay (baseDelay: 10s)
--- | --- | --- | ---
1 | 2s | 5s | 10s
2 | 4s | 10s | 20s
3 | 8s | 20s | 40s
4 | 16s | 40s | 80s
5 | 32s | 80s | 160s
6 | 64s | 160s | 320s
7 | 128s | 320s | 640s

### When Tasks Fail Permanently

Section titled “When Tasks Fail Permanently”

A task is marked as permanently failed when:

- It has been attempted `maxAttempts` times
- Each attempt resulted in an error
- The task status changes from `queued` to `failed`
- The error message from the last attempt is stored

## Worker Configuration

Section titled “Worker Configuration”

These settings control how the EdgeWorker processes tasks from your flows. They’re passed when starting the worker and affect performance and resource utilization.

Note

Worker configuration requires redeploying your Edge Function to change, while flow/step configuration can be updated on the fly with SQL queries. See Update Flow Options for details on updating configuration without redeployment.

## Default Worker Configuration

Section titled “Default Worker Configuration”

When using pgflow with EdgeWorker, configure the worker’s behavior:

    import { EdgeWorker } from '@pgflow/edge-worker';import MyFlow from './flows/my_flow.ts';
    EdgeWorker.start(MyFlow, {  // Worker concurrency settings  maxConcurrent: 10,      // Process up to 10 tasks simultaneously  maxPgConnections: 4,    // Database connection pool size
      // Polling configuration  batchSize: 10,          // Fetch up to 10 tasks per poll  maxPollSeconds: 2,      // Long-poll duration  pollIntervalMs: 100,    // Database polling frequency});

### `maxConcurrent`

Section titled “maxConcurrent”

Type: `number`
Default: `10`

Maximum number of tasks that can be processed simultaneously. Increase for I/O-heavy tasks, decrease for CPU-heavy tasks.

### `maxPgConnections`

Section titled “maxPgConnections”

Type: `number`
Default: `4`

Size of the PostgreSQL connection pool. Should generally be less than `maxConcurrent` since not all tasks need a connection at all times.

### `batchSize`

Section titled “batchSize”

Type: `number`
Default: `10`

Maximum number of tasks to fetch in a single poll operation. Larger batches are more efficient but may lead to uneven work distribution.

### `maxPollSeconds`

Section titled “maxPollSeconds”

Type: `number`
Default: `2`

How long to wait for tasks during each poll cycle. Shorter durations ensure more responsive task pickup.

Caution

Keep `maxPollSeconds` below the heartbeat interval (5 seconds) for flow workers to ensure proper heartbeat timing.

### `pollIntervalMs`

Section titled “pollIntervalMs”

Type: `number`
Default: `100`

How frequently (in milliseconds) to check the database for new tasks during the poll cycle.

## Accessing Worker Configuration in Handlers

Section titled “Accessing Worker Configuration in Handlers”

Your flow step handlers can access the resolved worker configuration through the context object. For detailed information about available configuration options and usage patterns, see Context.

## Complete Examples

Section titled “Complete Examples”

These examples show how flow/step configuration and worker configuration work together in real scenarios.

### High-Reliability API Integration

Section titled “High-Reliability API Integration”

For flows that interact with external APIs that may be temporarily unavailable:

    new Flow({  slug: 'sync_external_data',  maxAttempts: 7,     // More retries for transient failures  baseDelay: 10,      // Start with 10s, reaching ~20 minutes by attempt 7  timeout: 300        // 5 minutes for slow APIs})

### Fast Local Processing

Section titled “Fast Local Processing”

For flows that process data locally and should fail fast:

    new Flow({  slug: 'process_upload',  maxAttempts: 2,     // Fail fast - only 1 retry  baseDelay: 1,       // Minimal delay between attempts  timeout: 30         // Quick operations})

### Mixed Workload

Section titled “Mixed Workload”

When different steps have different reliability requirements:

    new Flow({  slug: 'data_pipeline',  maxAttempts: 3,     // Sensible defaults  baseDelay: 5,  timeout: 60})  .step({    slug: 'validate_input',    maxAttempts: 1,   // No retries - validation should not fail    timeout: 10       // Very quick  }, validateHandler)  .step({    slug: 'fetch_external',    maxAttempts: 5,   // External API might be flaky    baseDelay: 10,    // Longer delays for external service    timeout: 120      // Allow more time  }, fetchHandler)  .step({    slug: 'save_results',    // Use flow defaults  }, saveHandler)

### Multi-Day Workflow

Section titled “Multi-Day Workflow”

For workflows that span multiple days with scheduled follow-ups:

    new Flow({  slug: 'user_onboarding',  maxAttempts: 3,  baseDelay: 5,  timeout: 60})  .step({    slug: 'send_welcome_email',    // Executes immediately  }, sendWelcomeHandler)  .step({    slug: 'send_day_3_tips',    startDelay: 259200,  // 3 days (3 * 24 * 60 * 60)    timeout: 120  }, sendTipsHandler)  .step({    slug: 'send_week_review',    startDelay: 604800,  // 7 days after previous step    timeout: 120  }, sendReviewHandler)