Skip to content

# Array and Map Steps

pgflow’s map steps enable parallel processing of arrays by automatically creating multiple tasks - one for each array element. This guide explains how map steps work, when to use them, and their unique characteristics.

## Why Map Steps?

Section titled “Why Map Steps?”

When processing arrays in workflows, you often want to:

- Apply the same operation to each element independently
- Process items in parallel for better performance
- Transform data collections efficiently
- Make batch API calls or database queries

Map steps solve these needs by providing a declarative way to spawn parallel tasks from arrays.

## Mental Model: One Step, Many Tasks

Section titled “Mental Model: One Step, Many Tasks”

Both step types produce a single output, but achieve it differently:

Regular Step:

    Input → [Single Task] → Output

Map Step:

             ┌→ [Task 0] → Result 0 ┐         │→ [Task 1] → Result 1 │Array[5] │→ [Task 2] → Result 2 │→ Array[5]         │→ [Task 3] → Result 3 │         └→ [Task 4] → Result 4 ┘
    Each task:- Processes one element- Has own retry counter- Retries independently (but if exhausted, fails the entire step)

The key insight: Map steps transform arrays element-by-element through parallel execution, but still produce one step output (an array).

## How Map Steps Work

Section titled “How Map Steps Work”

### Task Spawning and Distribution

Section titled “Task Spawning and Distribution”

When a map step becomes ready, pgflow:

1. Validates the input is an array - Non-array inputs fail the step
2. Creates N tasks - One per array element, with `task_index` 0 to N-1
3. Distributes elements - Task 0 gets `array[0]`, task 1 gets `array[1]`, etc.
4. Queues tasks - All tasks are enqueued for parallel processing

    // This map step with 3 elements....array({ slug: 'userIds' }, () => ['user1', 'user2', 'user3']).map({ slug: 'userData', array: 'userIds' }, async (id) => {  // Task 0 receives: 'user1'  // Task 1 receives: 'user2'  // Task 2 receives: 'user3'  return await fetchUserData(id);})

### Independent Task Execution

Section titled “Independent Task Execution”

Each map task executes independently with its own:

- Retry counter: If one task fails, only that task retries, not the entire array
- Retry isolation: Other tasks continue while one retries

This independence is crucial for resilience:

    // Processing 100 API calls.map({ slug: 'apiCalls', array: 'items' }, async (item) => {  return await callAPI(item);  // If item #47 fails, only task #47 retries})
    // vs single step processing.step({ slug: 'apiCallsBatch' }, async (input) => {  // If ANY call fails, the ENTIRE step retries all 100 calls!  return await Promise.all(input.items.map(item => callAPI(item)));})

### Output Aggregation

Section titled “Output Aggregation”

As map tasks complete, their outputs are collected:

    // If the map tasks return:// Task 0: { name: 'Alice', id: 'user1' }// Task 1: { name: 'Bob', id: 'user2' }// Task 2: { name: 'Charlie', id: 'user3' }
    // The aggregated output becomes:[  { name: 'Alice', id: 'user1' },  { name: 'Bob', id: 'user2' },  { name: 'Charlie', id: 'user3' }]

Important: Order is preserved based on `task_index`, not completion time.

## Root Maps vs Dependent Maps

Section titled “Root Maps vs Dependent Maps”

Map steps operate in two modes:

### Root Maps

Section titled “Root Maps”

Process the flow’s input array directly by omitting the `array` property:

    // Flow input MUST be an arraynew Flow<string[]>({ slug: 'batch_processor' })  .map(    { slug: 'processEach' },  // No 'array' property    (item) => processItem(item)  );

Starting this flow:

    SELECT pgflow.start_flow(  flow_slug => 'batch_processor',  input => '["item1", "item2", "item3"]'::jsonb);

### Dependent Maps

Section titled “Dependent Maps”

Process another step’s array output by specifying the `array` property:

    new Flow<{ searchQuery: string }>({ slug: 'search_pipeline' })  .step(    { slug: 'searchResults' },    async (input) => await searchAPI(input.run.searchQuery)  )  .map(    { slug: 'processResults', array: 'searchResults' },  // Processes search output    (result) => extractRelevantData(result)  );

## Handler Differences

Section titled “Handler Differences”

The most important difference between regular steps and map steps is what the handler receives:

### Regular Step Handler

Section titled “Regular Step Handler”

    .step({ slug: 'regular' }, async (input) => {  // input contains:  // - input.run: The original flow input  // - input.dependencyName: Output from each dependency  return processAllData(input);})

### Map Step Handler

Section titled “Map Step Handler”

    .map({ slug: 'mapStep', array: 'source' }, async (item) => {  // item is ONLY the individual array element  // No access to input.run or other dependencies  return processItem(item);})

Limited to Array Elements

Map handlers cannot access `input.run` or other step outputs - they only receive their assigned array element. For now, if you need additional data, bundle it into the array elements in a previous step. In future versions, the flow input will be accessible via the handler’s context parameter.

## Edge Cases and Special Behaviors

Section titled “Edge Cases and Special Behaviors”

### Empty Array Cascade

Section titled “Empty Array Cascade”

When a map step receives an empty array (`[]`):

1. The step completes immediately without creating tasks
2. The step outputs an empty array `[]`
3. Dependent map steps also receive `[]` and complete immediately
4. This cascades through the entire chain in a single transaction

    // If fetch_items returns [].array({ slug: 'fetchItems' }, () => []).map({ slug: 'process', array: 'fetchItems' }, (item) => transform(item)).map({ slug: 'enrich', array: 'process' }, (item) => enrich(item))
    // Result: All three steps complete with []

### NULL Handling

Section titled “NULL Handling”

NULL values in arrays are preserved and distributed:

    // Array with NULL['item1', null, 'item3']
    // Task 0: 'item1'// Task 1: null// Task 2: 'item3'
    // If task 1 returns null, aggregated output:['result1', null, 'result3']

### Type Violations

Section titled “Type Violations”

When a single step outputs non-array data to a map step:

1. The SQL Core detects the type violation
2. The entire run is marked as failed
3. The invalid output is stored for debugging
4. All queued messages are archived to prevent orphaned tasks

    // This will fail the run:.step({ slug: 'single' }, () => ({ not: 'an array' })).map({ slug: 'map', array: 'single' }, (item) => item)  // Type violation!

## Common Patterns

Section titled “Common Patterns”

### ETL Pipeline

Section titled “ETL Pipeline”

    new Flow<{ source: string }>({ slug: 'etl' })  .step({ slug: 'extract' }, async (input) => {    return await fetchCSVRows(input.run.source);  })  .map({ slug: 'transform', array: 'extract' }, (row) => {    return {      ...row,      processed_at: new Date().toISOString(),      normalized: normalizeData(row)    };  })  .step({ slug: 'load', dependsOn: ['transform'] }, async (input) => {    // input.transform is the aggregated array    return await bulkInsertToDatabase(input.transform);  });

### Batch API Calls

Section titled “Batch API Calls”

    new Flow<{}>({ slug: 'notify_users' })  .step({ slug: 'users' }, async () => {    return await getActiveUsers();  })  .map({ slug: 'notificationResults', array: 'users' }, async (user) => {    return await sendEmail(user.email, 'Your weekly update');  })  .step({ slug: 'summary', dependsOn: ['notificationResults'] }, (input) => {    const successful = input.notificationResults.filter(r => r.success);    return {      sent: successful.length,      total: input.notificationResults.length    };  });

### Dynamic Expansion Pattern

Section titled “Dynamic Expansion Pattern”

Process a single input in multiple ways, determined at runtime:

    // Analyze product review for each mentioned featurenew Flow<{ review: string }>({ slug: 'review_analysis' })  .array(    { slug: 'features' },    async (input) => {      // Find which features are mentioned in the review      return await extractProductFeatures(input.run.review);      // Returns: ['battery_life', 'screen_quality', 'camera']    }  )  .array(    { slug: 'analysisPairs', dependsOn: ['features'] },    (input) => {      // Bundle review text with each feature for parallel analysis      return input.features.map(feature => ({        reviewText: input.run.review,        feature: feature      }));    }  )  .map(    { slug: 'sentimentAnalysis', array: 'analysisPairs' },    async (item) => ({      feature: item.feature,      sentiment: await analyzeSentiment(item.reviewText, item.feature),      quotes: await extractQuotes(item.reviewText, item.feature)    })  );

Why the pairing step?

Map handlers currently only receive individual array elements - they cannot access `input.run` or other dependencies. The `analysisPairs` step is needed to bundle the original review text with each feature. In future versions, the flow input will be accessible via the handler’s context parameter, eliminating the need for this extra step. Until then, we recommend this pattern for bundling context.

The pattern: Single input → Discover array dynamically → Enrich with context → Process in parallel

Common use cases:

- Support tickets: Ticket → find similar cases → pair each with original → evaluate solutions
- Document analysis: Text → extract topics → bundle with document → analyze each topic
- Image processing: Image → detect objects → add image URL to each → process regions
- Security logs: Log entry → find anomalies → pair with context → investigate each

This enables runtime parallelism - you discover how many operations to run only after analyzing the input.

### Chained Maps

Section titled “Chained Maps”

    new Flow<{}>({ slug: 'imagePipeline' })  .array({ slug: 'imageUrls' }, () => getImageUrls())  .map({ slug: 'download', array: 'imageUrls' },    async (url) => await downloadImage(url)  )  .map({ slug: 'resize', array: 'download' },    async (image) => await resizeImage(image, { width: 800 })  )  .map({ slug: 'upload', array: 'resize' },    async (image) => await uploadToS3(image)  );

## Limitations and Considerations

Section titled “Limitations and Considerations”

### Single Dependency Constraint

Section titled “Single Dependency Constraint”

Map steps can only depend on one array source:

    // NOT ALLOWED - multiple dependencies.map({  slug: 'invalid',  array: 'source1',  dependsOn: ['source2']  // Error: Can't have both!}, handler)

### Limited Input Scope

Section titled “Limited Input Scope”

Map handlers can’t access the original flow input (`input.run`) or other step outputs:

    // Problem: Need user ID in map handlernew Flow<{ userId: string }>({ slug: 'processItems' })  .array({ slug: 'items' }, () => getItems())  .map({ slug: 'process', array: 'items' }, (item) => {    // Can't access input.run.userId here!    // Solution: Include userId in array elements  });
    // Solution: Enrich array elements firstnew Flow<{ userId: string }>({ slug: 'processItems' })  .step({ slug: 'items' }, async (input) => {    const items = await getItems();    return items.map(item => ({ ...item, userId: input.run.userId }));  })  .map({ slug: 'process', array: 'items' }, (item) => {    // Now item includes userId    return processWithUser(item, item.userId);  });

### Performance Considerations

Section titled “Performance Considerations”

- Task Overhead: Each array element creates a separate task with queue overhead
- Database Load: N tasks mean N database transactions
- Memory Usage: Large arrays spawn many concurrent tasks
- Rate Limiting: Consider external API limits when making parallel calls

Best Practices: Map Steps vs Single-Step Array Processing

Use map steps when:

- Making individual API calls - each item gets its own retry counter
- Operations can fail independently - one failure doesn’t retry successful items
- You need parallel processing with failure isolation

Use single-step processing when:

- The API supports batch operations (one call for entire array)
- Doing pure transformations (formatting strings, calculations)
- The overhead of multiple tasks isn’t worth it for simple operations

Remember: Map steps create N tasks with N database transactions - consider chunking for very large arrays (>1000 items)

## Summary

Section titled “Summary”

Map steps provide powerful parallel array processing:

- Automatic Parallelization: One step spawns N parallel tasks
- Simple Handler API: Focus on transforming individual elements
- Order Preservation: Results maintain array element order
- Graceful Edge Cases: Empty arrays and NULLs handled correctly
- Type Safety: TypeScript ensures correct array types

Use map steps when you need to process collections efficiently, but remember their constraints around dependencies and context access. For complex scenarios requiring flow context, enrich your array elements before mapping.

For practical examples of using map steps, see the Batch Processing with Map Steps guide.