---
title: "Template: Web Scraper"
description: "Apify-based web scraper that stores results in Supabase -- fetch structured data from any website via Apify actors"
loadWhen: "Applying the web-scraper workflow template"
---

**Category:** Data Collection

**Platform Tools:** `apify` (run actor), `supabase` (insert results)

**Credentials Required:**

- `apify` -- Apify API token (bearer type)
- `my-database` -- Supabase project URL and service role key

---

## What This Workflow Does

Runs an Apify actor to scrape structured data from a URL, then stores the scraped results in a Supabase table. The actor and table names are configurable. Suitable for extracting product listings, job postings, contact information, or any structured web content.

---

## Input Schema

```typescript
z.object({
  actorId: z.string(),          // Apify actor ID (e.g., 'apify/web-scraper')
  startUrls: z.array(z.string()), // URLs to scrape
  tableName: z.string(),        // Supabase table to store results in
  maxItems: z.number().optional(), // Maximum number of items to collect (default: 100)
})
```

## Output Schema

```typescript
z.object({
  itemsScraped: z.number(),     // Total items scraped
  itemsInserted: z.number(),    // Items successfully inserted into Supabase
  runId: z.string(),            // Apify run ID for reference
})
```

---

## Workflow Code Pattern

```typescript
import type { WorkflowDefinition } from '@elevasis/sdk'
import { StepType } from '@elevasis/sdk'
import { platform, PlatformToolError } from '@elevasis/sdk/worker'
import { z } from 'zod'

const inputSchema = z.object({
  actorId: z.string(),
  startUrls: z.array(z.string()),
  tableName: z.string(),
  maxItems: z.number().optional(),
})
const outputSchema = z.object({
  itemsScraped: z.number(),
  itemsInserted: z.number(),
  runId: z.string(),
})

type Input = z.infer<typeof inputSchema>

export const webScraper: WorkflowDefinition = {
  config: {
    resourceId: 'web-scraper',
    name: 'Web Scraper',
    type: 'workflow',
    description: 'Scrapes structured data via Apify and stores in Supabase',
    version: '1.0.0',
    status: 'dev',
  },
  contract: { inputSchema, outputSchema },
  steps: {
    scrape: {
      id: 'scrape',
      name: 'Run Apify Actor',
      description: 'Execute the Apify actor and collect results',
      inputSchema,
      outputSchema: z.object({ items: z.array(z.unknown()), runId: z.string() }),
      handler: async (input) => {
        const { actorId, startUrls, maxItems } = input as Input
        const result = await platform.call({
          tool: 'apify',
          method: 'runActor',
          credential: 'apify',
          params: { actorId, input: { startUrls: startUrls.map(url => ({ url })), maxItems: maxItems ?? 100 } },
        }) as { items: unknown[]; runId: string }
        return { items: result.items, runId: result.runId }
      },
      next: { type: StepType.LINEAR, target: 'store' },
    },
    store: {
      id: 'store',
      name: 'Store Results',
      description: 'Insert scraped items into Supabase',
      inputSchema: z.object({ items: z.array(z.unknown()), runId: z.string(), tableName: z.string() }),
      outputSchema,
      handler: async (input, context) => {
        const { items, runId, tableName } = input as { items: unknown[]; runId: string; tableName: string }
        const data = items.map(item => ({ ...(item as Record<string, unknown>), scraped_at: new Date().toISOString() }))
        await platform.call({
          tool: 'supabase',
          method: 'insert',
          credential: 'my-database',
          params: { table: tableName, data },
        })
        context.logger.info('Stored scraped items', { count: data.length, table: tableName })
        return { itemsScraped: items.length, itemsInserted: data.length, runId }
      },
      next: null,
    },
  },
  entryPoint: 'scrape',
}
```

---

## Adaptation Notes

- **Credential names:** Replace `'apify'` and `'my-database'` with the credential names the user configured in the command center.
- **Table schema:** Ensure the Supabase table exists and has columns that match the scraped data fields. The agent should check `data/schema.ts` if it exists.
- **Actor selection:** Apify has hundreds of public actors. Common choices: `apify/web-scraper` (generic), `apify/cheerio-scraper` (fast HTML parsing), specialized actors for LinkedIn, Amazon, etc.
- **Error handling:** Add try/catch for `PlatformToolError` if partial failure tolerance is needed.
- **Skill adaptation:** For beginners, explain what Apify is and walk through actor selection before generating code.

---

**Last Updated:** 2026-02-26
