# AI Web Scraper

import { Steps, LinkCard, CardGrid } from '@astrojs/starlight/components';
import { FileTree } from '@astrojs/starlight/components';
import JoinCommunity from '@/components/JoinCommunity.astro';

**Build an AI-powered web scraper: fetch → analyze → store.**

<div>
  <a href="/analyze_website.gif" target="_blank" rel="noopener noreferrer">
    <img src="/analyze_website_small.gif" alt="Description of animation" />
  </a>
</div>

---

:::note[Tech Stack]
- **Supabase Edge Functions** - Serverless runtime for workflow execution
- **Deno** - JavaScript/TypeScript runtime environment
- **pgflow** - Postgres-native workflow engine
- **OpenAI API** - AI content analysis
:::

This tutorial uses **pgflow**, a Postgres-native workflow engine that manages DAG dependencies, state transitions, and execution flow directly in your database. It works with **Edge Worker**, a lightweight runner that executes your workflow tasks, handles retries, and reports results back to pgflow - all running within Supabase Edge Functions. Together, they let you build reliable, observable workflows without extra infrastructure.

:::note[Complete Source Code]
All code for this tutorial is available in the [pgflow-dev/ai-web-scraper](https://github.com/pgflow-dev/ai-web-scraper) GitHub repository.
:::

## What you'll build

You'll create a practical AI web scraper workflow that:

<Steps>
1. Grabs content from any webpage with built-in error handling
2. Uses GPT-4o to generate summaries and extract relevant tags
3. Runs multiple AI operations in parallel (cutting processing time in half)
4. Stores everything neatly in your Postgres database
5. Auto-retries when things go wrong (because APIs sometimes fail)
</Steps>

<img src="/analyze_website.svg" alt="AI Web Scraper Workflow" title="AI-powered web scraper workflow DAG" className="responsive-svg" />

## Project Structure

Here's the file structure we'll create:

<FileTree>
- supabase/
  - functions/
    - _tasks/
      - scrapeWebsite.ts
      - summarize.ts
      - extractTags.ts
      - saveToDb.ts
    - _flows/
      - analyze_website.ts
    - analyze_website_worker/
      - index.ts
</FileTree>

:::tip[Why pgflow for AI scraping?]
pgflow handles AI workflow complexity: automatic retries for API failures, parallel processing, and full observability.
Everything runs in your Supabase project - no external infrastructure needed.
:::

## Prerequisites

:::caution[Before you begin]

1. Have a Supabase project initialized locally - see [Supabase Local Development Guide](https://supabase.com/docs/guides/local-development)
   ```sh
   npx supabase init
   ```

2. [Install pgflow](/get-started/installation/) - it automatically sets up Edge Worker environment variables
   ```sh
   npx pgflow@latest install
   ```

3. Add your [OpenAI API key](https://platform.openai.com/api-keys) to `supabase/functions/.env`:
   ```diff
     # supabase/functions/.env
     EDGE_WORKER_DB_URL=postgres://...
     EDGE_WORKER_LOG_LEVEL=info
   + OPENAI_API_KEY=sk-...
   ```

   **Important:** Place this `.env` file in the `supabase/functions` directory, not in your project root. Edge Functions specifically look for environment variables in this location.
:::

### Versions Used in This Tutorial

This tutorial was tested with these specific tool versions:

| Tool | Tested version |
|------|----------------|
| Supabase CLI | 2.22.12 |
| pgflow CLI | 0.2.5 |
| Deno | 1.45.2 |

## What you'll learn

<Steps>
1. Write task functions to fetch and process web content
2. Generate structured data from AI using type-safe schemas
3. Create parallel DAG workflows with the TypeScript DSL
4. Compile flows to SQL and apply migrations
5. Execute workflows using the Edge Worker
</Steps>

## Get started

<CardGrid>
  <LinkCard
    title="Part 1: Build the Backend"
    description="Create the database schema, AI tasks, and workflow"
    href="/tutorials/ai-web-scraper/backend/"
  />
</CardGrid>

<JoinCommunity />