Skip to content

# AI Web Scraper

Build an AI-powered web scraper: fetch → analyze → store.

Tech Stack

- Supabase Edge Functions - Serverless runtime for workflow execution
- Deno - JavaScript/TypeScript runtime environment
- pgflow - Postgres-native workflow engine
- OpenAI API - AI content analysis

This tutorial uses pgflow, a Postgres-native workflow engine that manages DAG dependencies, state transitions, and execution flow directly in your database. It works with Edge Worker, a lightweight runner that executes your workflow tasks, handles retries, and reports results back to pgflow - all running within Supabase Edge Functions. Together, they let you build reliable, observable workflows without extra infrastructure.

Complete Source Code

All code for this tutorial is available in the pgflow-dev/ai-web-scraper GitHub repository.

## What you’ll build

Section titled “What you’ll build”

You’ll create a practical AI web scraper workflow that:

1. Grabs content from any webpage with built-in error handling
2. Uses GPT-4o to generate summaries and extract relevant tags
3. Runs multiple AI operations in parallel (cutting processing time in half)
4. Stores everything neatly in your Postgres database
5. Auto-retries when things go wrong (because APIs sometimes fail)

## Project Structure

Section titled “Project Structure”

Here’s the file structure we’ll create:

- Directory
  supabase/
  - Directory
    functions/
    - Directory
      _tasks/
      - scrapeWebsite.ts
      - summarize.ts
      - extractTags.ts
      - saveToDb.ts
    - Directory
      _flows/
      - analyze_website.ts
    - Directory
      analyze_website_worker/
      - index.ts

Why pgflow for AI scraping?

pgflow handles AI workflow complexity: automatic retries for API failures, parallel processing, and full observability. Everything runs in your Supabase project - no external infrastructure needed.

## Prerequisites

Section titled “Prerequisites”

Before you begin

1. Have a Supabase project initialized locally - see Supabase Local Development Guide

   Terminal window

       npx supabase init

2. Install pgflow - it automatically sets up Edge Worker environment variables

   Terminal window

       npx pgflow@latest install

3. Add your OpenAI API key to `supabase/functions/.env`:

   supabase/functions/.env

       EDGE_WORKER_DB_URL=postgres://...EDGE_WORKER_LOG_LEVEL=infoOPENAI_API_KEY=sk-...

   Important: Place this `.env` file in the `supabase/functions` directory, not in your project root. Edge Functions specifically look for environment variables in this location.

### Versions Used in This Tutorial

Section titled “Versions Used in This Tutorial”

This tutorial was tested with these specific tool versions:

Tool | Tested version
--- | ---
Supabase CLI | 2.22.12
pgflow CLI | 0.2.5
Deno | 1.45.2

## What you’ll learn

Section titled “What you’ll learn”

1. Write task functions to fetch and process web content
2. Generate structured data from AI using type-safe schemas
3. Create parallel DAG workflows with the TypeScript DSL
4. Compile flows to SQL and apply migrations
5. Execute workflows using the Edge Worker

## Get started

Section titled “Get started”
Part 1: Build the Backend Create the database schema, AI tasks, and workflow

## Join the Community

Connect on Discord

Have questions or need help? pgflow is just getting started - join us on Discord to ask questions, share feedback, or discuss partnership opportunities.

##### Join Discord →