Skip to content

# Batch Process with Map Steps

This guide shows practical patterns for using map steps to process collections of data in parallel, from simple transformations to complex batch operations.

## Processing a List of URLs

Section titled “Processing a List of URLs”

A common use case is fetching and processing multiple web pages in parallel.

### Basic URL Processing

Section titled “Basic URL Processing”

    import { Flow } from '@pgflow/dsl/supabase';
    const ScrapeMultipleUrls = new Flow<string[]>({  // Flow input must be array for root map  slug: 'scrape_multiple_urls',  maxAttempts: 3,})  .map(    { slug: 'scrapedPages' },    async (url) => await scrapeWebpage(url)  )  .step(    { slug: 'summary', dependsOn: ['scrapedPages'] },    (input) => summarizeResults(input.scrapedPages)  );
    // Usage (SQL):SELECT pgflow.start_flow(  flow_slug => 'scrape_multiple_urls',  input => '["https://example.com", "https://example.org", "https://example.net"]'::jsonb);

## Batch API Calls

Section titled “Batch API Calls”

Make multiple API calls in parallel while respecting rate limits.

### Simple Batch API Calls

Section titled “Simple Batch API Calls”

    const BatchApiCalls = new Flow<string[]>({  // Array input for root map  slug: 'batch_api_calls',})  .map(    { slug: 'userData' },    async (userId) => await fetchUserData(userId)  )  .step(    { slug: 'summary', dependsOn: ['userData'] },    (input) => generateUserReport(input.userData)  );

## Data Transformation Pipeline

Section titled “Data Transformation Pipeline”

Transform and validate data collections efficiently.

### CSV Processing Pipeline

Section titled “CSV Processing Pipeline”

    const CsvProcessor = new Flow<{ csvUrl: string }>({  slug: 'csv_processor',})  .step(    { slug: 'csvRows' },    async (input) => await fetchAndParseCSV(input.run.csvUrl)  )  .map(    { slug: 'validatedRows', array: 'csvRows' },    (row) => validateRow(row)  )  .map(    { slug: 'processedRows', array: 'validatedRows' },    (row) => transformIfValid(row)  )  .step(    { slug: 'saveResults', dependsOn: ['processedRows'] },    async (input, context) => await saveProcessedData(input.processedRows, context)  );

## Combining Results with Aggregation

Section titled “Combining Results with Aggregation”

Process items individually then aggregate results for analysis.

### Statistical Analysis

Section titled “Statistical Analysis”

    const StatisticalAnalysis = new Flow<{ experimentId: string }>({  slug: 'statistical_analysis',})  .step({ slug: 'samples' }, async (input, context) => {    // Fetch measurements from database    const { data } = await context.supabase      .from('measurements')      .select('*')      .eq('experiment_id', input.run.experimentId);    return data;  })  .map(    { slug: 'analyzedSamples', array: 'samples' },    (sample) => analyzeSample(sample)  )  .step(    { slug: 'saveStatistics', dependsOn: ['analyzedSamples'] },    async (input, context) => {      const stats = calculateStatistics(input.analyzedSamples);
          // Save analysis results to database      await context.supabase        .from('experiment_statistics')        .insert({          experiment_id: input.run.experimentId,          stats,          analyzed_at: new Date().toISOString()        });
          return stats;    }  );

### Notification Batching

Section titled “Notification Batching”

    const NotificationBatcher = new Flow<{}>({  slug: 'notification_batcher',})  .step({ slug: 'recipients' }, async (input, context) => {    const { data } = await context.supabase      .from('users')      .select('id, email, preferences')      .eq('notifications_enabled', true);    return data;  })  .map(    { slug: 'sendNotifications', array: 'recipients' },    async (user) => {      try {        // Send email notification        await sendEmail({          to: user.email,          subject: 'Weekly Update',          template: 'weekly_update',          data: { userId: user.id }        });
            return {          userId: user.id,          success: true,          sentAt: new Date().toISOString()        };      } catch (error) {        return {          userId: user.id,          success: false,          error: error.message        };      }    }  )  .step(    { slug: 'updateStatus', dependsOn: ['sendNotifications'] },    async (input, context) => {      const results = input.sendNotifications;
          // Update last notification timestamp for successful sends      const successful = results.filter(r => r.success);      if (successful.length > 0) {        await context.supabase          .from('users')          .update({ last_notification: new Date().toISOString() })          .in('id', successful.map(r => r.userId));      }
          // Log failures      const failed = results.filter(r => !r.success);      if (failed.length > 0) {        await context.supabase          .from('notification_failures')          .insert(failed);      }
          return {        sent: successful.length,        failed: failed.length,        total: results.length      };    }  );

## Debugging Map Steps

Section titled “Debugging Map Steps”

When debugging map steps, you need to understand how individual tasks behave.

### Logging Individual Task Progress

Section titled “Logging Individual Task Progress”

    const DebugFlow = new Flow<{ items: string[] }>({  slug: 'debug_flow',})  .map(    { slug: 'processWithLogging' },    async (item, context) => {      const taskIndex = context.stepTask.task_index;
          console.log(`Task ${taskIndex}: Starting to process "${item}"`);
          try {        const result = await processItem(item);        console.log(`Task ${taskIndex}: Successfully processed "${item}"`);        return { success: true, item, result };      } catch (error) {        console.error(`Task ${taskIndex}: Failed to process "${item}":`, error);        return { success: false, item, error: error.message };      }    }  );

### Checking Task Execution Order

Section titled “Checking Task Execution Order”

Remember that map tasks execute in parallel but outputs are aggregated in order:

    const OrderTest = new Flow<{ items: number[] }>({  slug: 'order_test',})  .map(    { slug: 'processWithDelay' },    async (item, context) => {      // Longer delay for lower indices to test order preservation      const delay = (10 - item) * 100;      await new Promise(resolve => setTimeout(resolve, delay));
          return {        item,        taskIndex: context.stepTask.task_index,        processedAt: new Date().toISOString()      };    }  )  .step(    { slug: 'verifyOrder', dependsOn: ['processWithDelay'] },    (input) => {      // Output array maintains original order despite different completion times      const inOrder = input.processWithDelay        .every((result, index) => result.taskIndex === index);
          return {        orderMaintained: inOrder,        results: input.processWithDelay      };    }  );

## Common Gotchas and Solutions

Section titled “Common Gotchas and Solutions”

### Passing Additional Context

Section titled “Passing Additional Context”

Since map handlers only receive individual elements, you need to include context in the array:

    // Problem: Need configuration in map handlerconst ProblemFlow = new Flow<{ apiKey: string, ids: string[] }>({  slug: 'problem_flow',})  .map({ slug: 'fetch' }, async (id) => {    // Can't access input.run.apiKey here!    return await fetchWithKey(id, ???);  // No access to apiKey  });
    // Solution: Enrich array elements firstconst SolutionFlow = new Flow<{ apiKey: string, ids: string[] }>({  slug: 'solution_flow',})  .step(    { slug: 'prepareItems' },    (input) => {      // Include needed context in each element      return input.run.ids.map(id => ({        id,        apiKey: input.run.apiKey      }));    }  )  .map(    { slug: 'fetch', array: 'prepareItems' },    async (item) => {      // Now we have access to both id and apiKey      return await fetchWithKey(item.id, item.apiKey);    }  );

### Handling Empty Arrays

Section titled “Handling Empty Arrays”

Empty arrays cascade through map steps automatically:

    const EmptyHandling = new Flow<{}>({  slug: 'empty_handling',})  .step({ slug: 'maybeEmpty' }, async () => {    const results = await fetchData();    return results || [];  // Might return []  })  .map(    { slug: 'process', array: 'maybeEmpty' },    (item) => processItem(item)  )  .step(    { slug: 'handleResults', dependsOn: ['process'] },    (input) => {      if (input.process.length === 0) {        return { message: 'No items to process' };      }      return { processed: input.process.length };    }  );

## Summary

Section titled “Summary”

Map steps provide powerful parallel processing capabilities for pgflow workflows:

- URL Processing: Fetch and process multiple web pages concurrently
- API Batching: Make parallel API calls with rate limiting
- Data Pipelines: Transform and validate collections efficiently
- Result Aggregation: Process individually, analyze collectively

Remember to:

- Include necessary context in array elements
- Handle errors gracefully at the individual task level
- Consider rate limits and resource constraints
- Use chunking for very large arrays
- Monitor task execution for debugging

For more details on how map steps work internally, see the Array and Map Steps concepts guide.