# web-scraper-seo

A Claude Code skill that scrapes websites, downloads their assets, analyzes SEO techniques, and produces structured output for mockup generation and web development brainstorming. Scraped content can be served locally and verified with Playwright.

## What It Does

Give it a URL (or a list of URLs) and an optional crawl depth, and it will:

1. **Crawl** the site breadth-first, following navigation links up to N levels deep
2. **Render JS** (optional `--render-js`) — use Playwright to get the fully rendered DOM for SPAs before processing
3. **Discover pages** (optional `--sitemap`) — fetch sitemap.xml to find pages not linked from navigation
4. **Download** all assets — CSS, JS, images, fonts — including assets referenced inside CSS files (`@import`, `url()`)
5. **Rewrite** HTML and CSS so all asset paths point to local copies
6. **Analyze** every page for SEO metadata, heading structure, structured data, link patterns, image optimization, and technical signals
7. **Detect broken links** — cross-references failed URLs with the pages that link to them, producing `broken-links.json`
8. **Generate** a `site-report.json` manifest with the full sitemap tree, asset inventory, design system extraction (colors, fonts, layout, framework detection), and SEO highlights
9. **Checkpoint** — saves state after every page so interrupted crawls can `--resume`
10. **Serve** the scraped site locally on an HTTP server so you can browse it or let Claude view it with Playwright
11. **Screenshot** the served pages at multiple viewports (desktop, tablet, mobile) for visual reference

The end result is a local copy of the site that Claude can read, reference, and use as a foundation to recreate pages, remix designs, apply SEO patterns to new projects, or compare multiple sites side-by-side.

## Installation

Drop the `web-scraper-seo.skill` file into your Claude Code skills directory, or copy the `web-scraper-seo/` folder directly.

### Dependencies

The skill installs these automatically on first run:

```
requests
beautifulsoup4
lxml
cssutils
playwright (optional — for screenshots and JS-rendered scraping)
```

If you want Playwright support, you'll also need the browser binary:

```bash
playwright install chromium
```

## Usage

### Basic — Single Page

> "Scrape https://example.com and show me the SEO analysis."

This scrapes just the one page (depth 0), downloads its assets, and produces the full analysis.

### Multi-Level Crawl

> "Scrape https://example.com with 2 levels of depth so I get the navigation structure."

Follows links found inside `<nav>`, `<header>`, and menu-role elements — not every link on the page — up to the specified depth.

### Multiple URLs

> "Scrape these sites and compare their SEO: https://site-a.com, https://site-b.com"

Scrapes each URL independently and produces separate analyses for comparison.

### Design Cloning

> "I want to clone the look and feel of https://cool-startup.com for a mockup. Scrape it with depth 1."

After scraping, Claude reads the `site-report.json` to understand the design system (colors, fonts, layout approach, framework) and can use the downloaded HTML/CSS as a starting point for new pages.

## CLI Reference

### scrape_site.py — Main Scraper

```bash
python scripts/scrape_site.py \
  --urls "https://example.com"            \  # Required. Comma-separated list.
  --depth 1                                \  # Optional. Default: 0 (single page).
  --output /path/to/output                 \  # Required. Where to save everything.
  --rate-limit 1.0                         \  # Optional. Seconds between requests. Default: 1.0
  --render-js                              \  # Optional. Use Playwright for JS rendering.
  --sitemap                                \  # Optional. Fetch sitemap.xml for page discovery.
  --resume                                 \  # Optional. Resume interrupted crawl from checkpoint.
  --proxy "socks5://proxy:1080"            \  # Optional. HTTP/HTTPS/SOCKS5 proxy for all requests.
  --cookies "session=abc;token=xyz"        \  # Optional. Auth cookies (inline or path to JSON file).
  --headers "Authorization:Bearer tok123"  \  # Optional. Custom HTTP headers (semicolon-separated).
  --basic-auth "user:password"                # Optional. HTTP Basic Auth credentials.
```

### serve_site.py — Local Server

```bash
python scripts/serve_site.py \
  --directory /path/to/scraped-site        \  # Required. Output from scrape_site.py.
  --port 8080                              \  # Optional. Default: 8080. Auto-finds next open port.
  --host 0.0.0.0                           \  # Optional. Default: 0.0.0.0
  --background                             \  # Optional. Run as background thread.
  --live-reload                            \  # Optional. Auto-refresh browser on file changes.
  --https                                     # Optional. Serve over HTTPS with self-signed cert.
```

The server maps `/` to `pages/index.html`, `/about` to `pages/about.html`, and `/assets/*` directly. It adds CORS headers and writes a `.server.pid` file for cleanup. With `--live-reload`, a JS snippet is injected into HTML responses to auto-refresh when files change. With `--https`, a self-signed TLS certificate is generated via `openssl`.

### playwright_view.py — Screenshots & JS Rendering

Four subcommands:

```bash
# Single screenshot
python scripts/playwright_view.py screenshot \
  --url "http://localhost:8080"             \
  --output screenshot.png                  \
  --viewport 1280x800                      \  # Optional. Default: 1280x800
  --full-page                              \  # Optional. Captures entire scrollable page.
  --wait 2                                 \  # Optional. Seconds to wait after load.
  --device "iPhone 13"                        # Optional. Emulate a specific device.

# Multi-viewport responsive screenshots
python scripts/playwright_view.py responsive \
  --url "http://localhost:8080"             \
  --output-dir /path/to/screenshots           # Produces desktop_1280x800.png,
                                              #   tablet_768x1024.png, mobile_375x812.png

# Extract JS-rendered HTML (for SPAs)
python scripts/playwright_view.py render \
  --url "https://spa-site.com"             \
  --output rendered.html                   \
  --wait 3                                    # Longer wait for complex JS apps.

# Crawl with JS rendering
python scripts/playwright_view.py js-crawl \
  --urls "https://spa-site.com"            \
  --output-dir /path/to/rendered           \
  --depth 1                                \
  --wait 2
```

### seo_analyzer.py — Standalone SEO Analysis

Run deeper analysis on already-scraped content:

```bash
python scripts/seo_analyzer.py \
  --input /path/to/scraped-site            \  # Required. Output from scrape_site.py.
  --format json                               # Optional. 'json' (default) or 'yaml'.
```

Produces `seo/seo-analysis.json` (or `.yaml`) with per-page scoring (0–100, graded A–F), heading hierarchy validation, meta tag quality assessment, link pattern analysis, image optimization audit, and aggregated top issues.

### extract_tokens.py — Design Token Extractor

Extract structured design tokens from scraped CSS into multiple output formats:

```bash
python scripts/extract_tokens.py \
  --input /path/to/scraped-site            \  # Required. Output from scrape_site.py.
  --format all                                # Optional. 'css', 'json', 'tailwind', or 'all' (default).
```

Reads all CSS sources (external stylesheets, inline `<style>`, `style=""` attributes) and produces `design-tokens/tokens.css` (CSS custom properties), `design-tokens/tokens-w3c.json` (W3C draft spec), `design-tokens/tailwind.config.js` (theme.extend snippet), and `design-tokens/tokens-raw.json` (raw extracted values). Extracts colors (primary/neutral/accent), typography scales, spacing systems, border radii, box shadows, and breakpoints.

### a11y_analyzer.py — Accessibility Audit

Run a WCAG 2.1 Level AA audit on scraped pages:

```bash
python scripts/a11y_analyzer.py \
  --input /path/to/scraped-site            \  # Required. Output from scrape_site.py.
  --format json                            \  # Optional. 'json' (default).
  --page /path/to/single-file.html            # Optional. Analyze one HTML file instead of entire site.
```

Checks 12 WCAG criteria (language, title, headings, alt text, form labels, link purpose, ARIA validity, focus indicators, skip navigation, landmarks, table headers, color contrast). Each page receives a 0–100 score with letter grade (A–F). Output saved to `a11y/a11y-analysis.json`, `a11y/a11y-summary.json`, and `a11y/per-page/`.

## Output Structure

```
scraped-sites/example.com/
├── site-report.json              # Master manifest — start here
├── broken-links.json             # Broken link report (source pages + anchor text)
├── redirect-chains.json          # Redirect chain data (hops, status codes)
├── .crawl-checkpoint.json        # Checkpoint for --resume (deleted on success)
├── pages/
│   ├── index.html                # Rewritten HTML with local asset paths
│   ├── about.html
│   └── products/
│       └── index.html
├── assets/
│   ├── css/                      # Downloaded stylesheets (url() paths rewritten)
│   ├── js/                       # Downloaded scripts
│   ├── images/                   # Downloaded images, favicons, icons
│   └── fonts/                    # Downloaded web fonts
├── seo/
│   ├── seo-summary.json          # Aggregated SEO findings
│   ├── seo-analysis.json         # Deep analysis with scores (from seo_analyzer.py)
│   └── per-page/
│       ├── index.json            # Per-page SEO data (including keyword density)
│       └── about.json
├── a11y/                          # Generated by a11y_analyzer.py
│   ├── a11y-summary.json         # Aggregated accessibility findings
│   ├── a11y-analysis.json        # Full analysis with per-page scores
│   └── per-page/
│       ├── index.json            # Per-page WCAG checks
│       └── about.json
└── design-tokens/                 # Generated by extract_tokens.py
    ├── tokens-raw.json           # Raw extracted values
    ├── tokens-w3c.json           # W3C Design Tokens format
    ├── tokens.css                # CSS custom properties
    └── tailwind.config.js        # Tailwind theme config
```

## site-report.json

The master manifest that ties everything together. Key sections:

| Section | What's In It |
|---------|-------------|
| `metadata` | Scrape date, URLs requested, depth, page/asset counts, duration, error count |
| `sitemap` | Hierarchical tree of pages with URLs, local paths, titles, depth, and children |
| `assets` | Lists of local paths grouped by type (css, js, images, fonts) |
| `seo.highlights` | SEO techniques detected (structured data, OG tags, etc.) |
| `seo.issues` | Common SEO problems found across pages |
| `design_notes.primary_colors` | Hex colors extracted from CSS |
| `design_notes.fonts_used` | Font families found (excluding generic fallbacks) |
| `design_notes.layout_approach` | CSS Grid, Flexbox, or traditional |
| `design_notes.responsive` | Whether `@media` queries are present |
| `design_notes.framework_detected` | Next.js, React, Angular, Vue, WordPress, etc. |
| `errors` | Any URLs that failed with status codes or error messages |

## SEO Analysis Coverage

### What Gets Extracted Per Page

**Meta & Content:**
Title tag (content + length), meta description (content + length), canonical URL, robots directives, Open Graph tags, Twitter Card tags, hreflang tags, word count.

**Structure:**
Full heading hierarchy (H1–H6) with text content, heading level validation (skipped levels, multiple H1s).

**Structured Data:**
JSON-LD blocks with detected schema.org types and full data.

**Links:**
Internal and external link counts, anchor text distribution, rel attributes (nofollow, sponsored, ugc), unique internal link targets.

**Images:**
Total count, alt text coverage (present / empty / missing), lazy loading usage, explicit dimension usage (CLS impact).

**Technical:**
Viewport meta, charset, favicon, inline vs external styles/scripts, async/defer script counts, render-blocking script count, resource hints (preconnect, prefetch, preload).

### SEO Scoring (seo_analyzer.py)

Each page receives a score out of 100 with letter grade (A–F):

| Category | Max Deduction | Triggers |
|----------|--------------|----------|
| Title tag | -15 | Missing (-15), too short/long (-5) |
| Meta description | -10 | Missing (-10), too short/long (-3) |
| Heading hierarchy | -15 | Per issue: missing H1, multiple H1s, skipped levels |
| Image optimization | -10 | Poor alt coverage (-10), fair (-5) |
| Technical signals | -15 | Per issue: missing viewport, blocking scripts, etc. |
| Structured data | +5 bonus | JSON-LD present |
| Open Graph | +3 bonus | OG tags present |

## Crawling Behavior

**Link Discovery:** The scraper only follows links found inside navigation-related elements: `<nav>`, `<header>`, `[role="navigation"]`, `[role="menu"]`, and common CSS classes (`.nav`, `.navbar`, `.menu`, `.main-menu`, `.primary-menu`, `.site-nav`, `.header-nav`, etc.). If no navigation elements are found, it falls back to all `<a>` tags but filters to HTML-like extensions only.

**Cycle Prevention:** A visited-URL set prevents re-scraping the same page. Fragment identifiers are stripped before comparison.

**Domain Scoping:** Only follows links to the same domain(s) as the input URLs.

**Rate Limiting:** 1 second between requests by default, configurable via `--rate-limit`.

**robots.txt:** Checked once per domain before crawling begins. Disallowed paths are skipped.

**User-Agent:** Identifies honestly as `ClaudeWebScraper/1.0` — does not impersonate browsers.

**Error Handling:** 30-second timeout per request, one retry on timeout, SSL verification fallback, skips and logs failures without stopping the crawl. Max asset size: 50 MB.

## Gaps

This section documents every known gap in the skill's current implementation. Each item describes what is missing, why it matters, and where applicable, what workaround exists today. The gaps are grouped by functional area.

> **Note:** Gaps #1–#27, #30, and #33–#34 have been resolved (some partially — #6 proxy rotation only, #14 data URI counting only, #23 no FID/INP). Only gaps #28 (diff/compare), #29 (HTML-to-framework), #31 (Figma), and #32 (screenshot diff) remain. JS rendering is integrated into the main crawl loop, same-domain iframe content is fetched and saved locally with shadow DOM flattened in JS mode, cookie/consent overlays are automatically dismissed, sitemap.xml parsing discovers orphan pages, authentication is supported via cookies/headers/basic-auth, proxy routing is available for all requests, checkpoint/resume prevents data loss on interruption, broken link detection produces actionable reports, keyword density analysis produces top-20 keywords with prominence detection, redirect chains are recorded with hop counts and status codes, internal link rewriting works across all pages, inline `<style>` processing extracts design patterns (including `@font-face`), a comprehensive WCAG 2.1 accessibility audit is available, and design tokens can be extracted in CSS custom properties, W3C JSON, and Tailwind config formats.

---

### Crawling & Scraping

**1. ~~No JavaScript-rendered scraping in the main crawl loop~~ RESOLVED**
The `--render-js` flag on `scrape_site.py` now uses Playwright to load each page in a headless Chromium browser, wait for `networkidle`, and extract the fully rendered DOM before passing it to the standard BeautifulSoup pipeline. Asset downloading, path rewriting, SEO analysis, and `site-report.json` generation all work identically — the only difference is the HTML source (Playwright-rendered vs. HTTP response). Falls back gracefully to HTTP-only if Playwright is not installed.

**2. ~~No handling of `<iframe>` content or shadow DOM elements~~ RESOLVED**
Same-domain `<iframe>` content is now fetched (via HTTP or Playwright depending on mode), saved to `pages/_iframes/iframe_<hash>.html`, and the parent page's `<iframe src>` is rewritten to point to the local copy. Cross-origin iframes are skipped (browser security prevents fetching them). The original URL is preserved in a `data-original-src` attribute. In `--render-js` mode, shadow DOM content is flattened into the page HTML before extraction: a JS snippet walks all shadow roots and inserts `<template shadowrootmode="open">` elements with the shadow tree's innerHTML, making the content visible to BeautifulSoup's parser. Note: shadow DOM flattening is best-effort and only works in `--render-js` mode.

**3. ~~No sitemap.xml parsing~~ RESOLVED**
The `--sitemap` flag fetches and parses `sitemap.xml` and `sitemap_index.xml` (including recursive sitemap indexes up to 3 levels deep). Discovered URLs are added to the BFS queue at depth 0 alongside navigation-discovered URLs. Handles XML namespace variations and reports the number of URLs discovered. For content-heavy sites, this finds pages not linked from navigation (orphan pages, blog posts, utility pages).

**4. ~~No cookie consent / popup dismissal~~ RESOLVED**
In `--render-js` mode, the scraper now automatically dismisses cookie consent banners, GDPR modals, and similar overlays after page load. It tries a prioritized list of selectors for named consent platforms (OneTrust, CookieBot, Osano, Quantcast, Complianz, Cookie Consent by Insites, Termly) followed by generic patterns matching `cookie`/`consent`/`accept` in element IDs, classes, and ARIA labels. As a final fallback, a z-index heuristic detects high-z overlays with dismiss buttons. The Playwright screenshot tools (`screenshot`, `responsive`, `render`) also dismiss overlays automatically. In HTTP-only mode (without `--render-js`), JS-injected banners don't appear in the HTML, so dismissal is not needed.

**5. ~~No authentication support (login-protected pages)~~ RESOLVED**
Three authentication mechanisms are now supported via CLI flags: `--cookies` injects session cookies (inline `key=value;key=value` or a path to a JSON file), `--headers` injects custom HTTP headers (e.g., `Authorization:Bearer token123`), and `--basic-auth user:password` sets HTTP Basic Auth credentials. All three apply to both the `requests.Session` (HTTP mode) and the Playwright browser context (`--render-js` mode). Cookies are injected per allowed domain in Playwright, and `http_credentials` is used for Basic Auth. Note: OAuth flows and form-based login are not automated — you must obtain tokens/cookies externally and pass them in.

**6. ~~No proxy or rotation support~~ PARTIALLY RESOLVED**
A `--proxy` flag now routes all requests through an HTTP, HTTPS, or SOCKS5 proxy (e.g., `--proxy socks5://proxy:1080`). The proxy is applied to both the `requests.Session` and the Playwright browser launch (via `proxy.server`). This enables geographic routing and basic IP masking. Note: proxy *rotation*, IP rotation, User-Agent rotation, and anti-detection techniques (Cloudflare bypass, etc.) are not implemented — only a single static proxy is supported.

**7. ~~No resume/checkpoint capability~~ RESOLVED**
The scraper now writes a `.crawl-checkpoint.json` file after every page is scraped. This checkpoint contains the full crawl state: visited URLs, downloaded asset mappings, page metadata, errors, link sources, and the remaining BFS queue. Using `--resume` loads this checkpoint and continues from where the crawl left off. On successful completion, the checkpoint file is automatically deleted. Checkpoint version is tracked for forward compatibility.

---

### Asset Handling

**8. ~~No inline `<style>` extraction~~ RESOLVED**
Inline `<style>` tags and `style=""` attributes are now processed: `url()` references are extracted, assets downloaded, and paths rewritten to local copies. The `extract_design_notes()` method scans both external CSS files and inline `<style>` blocks for colors, fonts, and layout patterns via the shared `_extract_css_design_patterns()` helper.

**9. ~~No CSS source map downloading~~ RESOLVED**
After downloading a CSS file, `download_asset()` calls `_download_source_map()` which searches for `/*# sourceMappingURL=... */` comments in the file content. If a source map reference is found (and is not a data URI), the `.css.map` file is downloaded to `assets/maps/`. This allows tracing minified CSS back to original SCSS/LESS/Stylus sources.

**10. ~~No handling of `@font-face` declarations inside inline `<style>` blocks~~ RESOLVED**
Resolved as a side effect of gap #8 (inline style processing). The `extract_css_assets()` method processes all `url()` references in inline `<style>` content, including `@font-face src: url(...)` declarations. Font files (.woff, .woff2, .ttf, .eot, .otf) are correctly categorized and downloaded to `assets/fonts/`. The `_rewrite_css_urls()` method rewrites these URLs to local paths in the saved HTML.

**11. ~~No JavaScript source map downloading~~ RESOLVED**
Resolved alongside gap #9. After downloading a JS file, `_download_source_map()` searches for `//# sourceMappingURL=...` comments. If found, the `.js.map` file is downloaded to `assets/maps/`.

**12. ~~No video or audio asset downloading~~ RESOLVED**
`extract_asset_urls()` now finds `<video src>`, `<audio src>`, and `<source src>` elements inside both `<video>` and `<audio>` containers. Media files are downloaded to `assets/media/` and paths rewritten in the HTML. Common media extensions (`.mp4`, `.webm`, `.ogg`, `.mp3`, `.wav`, `.m4a`, `.flac`) are mapped to the `media` category. The 50 MB per-file size limit still applies.

**13. ~~No SVG sprite handling~~ RESOLVED**
`extract_asset_urls()` now finds `<use href>` and `<use xlink:href>` elements, strips the `#fragment` to get the SVG file URL, and downloads the sprite sheet to `assets/images/`. During `rewrite_html()`, the `href`/`xlink:href` attributes are rewritten to point to the local copy while preserving the `#fragment` identifier (e.g., `sprites.svg#icon-name` → `../assets/images/sprites.svg#icon-name`). Inline SVG sprites (already in the HTML) are preserved as-is.

**14. ~~No data URI extraction or conversion~~ PARTIALLY RESOLVED**
Data URIs are still left inline (they are self-contained and work correctly). However, `extract_asset_urls()` now counts `data:` src attributes encountered across all pages, and the total is included in `site-report.json` under `metadata.data_uri_count`. This provides an accurate inventory of all resources used by the site. Note: data URIs are not extracted to separate files or converted — they remain inline.

**15. ~~No deduplication of identical assets served from different URLs~~ RESOLVED**
After downloading each asset, `download_asset()` computes a SHA-256 hash of the file content. If the hash matches a previously downloaded asset, the new file is deleted and the URL is mapped to the existing local path. This means `logo.png?v=1` and `logo.png?v=2` (if identical content) produce a single file on disk. The `downloaded_assets` dict correctly maps both URLs to the shared local path. Asset counts in `site-report.json` reflect unique files, not duplicate URLs.

---

### SEO Analysis

**16. ~~No page speed / performance scoring~~ RESOLVED**
In `--render-js` mode, `_fetch_with_playwright()` now collects performance metrics from the browser's Performance API after each page load. Metrics captured per page: `ttfb_ms` (Time to First Byte), `dom_interactive_ms`, `dom_content_loaded_ms`, `load_event_ms`, `lcp_ms` (Largest Contentful Paint), `lcp_element` (tag name), `cls` (Cumulative Layout Shift), `resource_count`, and `total_transfer_kb`. These are stored in each page's SEO data under `seo.performance` and included in per-page JSON files. Note: metrics are only available in `--render-js` mode; HTTP-only scraping cannot measure browser performance.

**17. ~~No mobile-friendliness testing beyond checking for viewport meta~~ RESOLVED**
`seo_analyzer.py` now includes `analyze_mobile_friendliness()` which checks each page for: tap target sizing (elements < 44px), text readability (font-size < 12px), horizontal overflow risk (fixed widths > 500px), mobile-hostile elements (`<embed>`, `<object>`, `<applet>`), and layout tables without headers. Each page receives a mobile score (0-100, graded A-F). Results are included in `seo/seo-analysis.json` under `mobile_friendliness` per page. Note: these are heuristic CSS-based checks on inline styles, not real mobile rendering tests.

**18. ~~No broken link detection~~ RESOLVED**
The scraper now tracks link sources during both navigation link extraction (`extract_nav_links`) and full link analysis (`analyze_seo`). A `link_sources` dict maps each target URL to the pages that link to it (with anchor text). After crawling, `generate_broken_links_report()` cross-references failed/missing URLs with their source pages and writes `broken-links.json` — sorted by impact (most referring pages first). The report shows: broken URL, error type, linking pages, and anchor text used.

**19. ~~No duplicate content detection across pages~~ RESOLVED**
After crawling, `detect_duplicate_content()` runs a two-pass analysis: (1) MD5 hashes of body text detect **exact duplicates** (identical content at different URLs), and (2) 4-word shingle sets with **Jaccard similarity > 80%** detect **near-duplicates**. Results are saved to `duplicate-content.json` with `exact_duplicates` (groups of identical URLs) and `near_duplicates` (pairs with similarity percentages). Also reported in `site-report.json` under `metadata.duplicate_pages_found`.

**20. ~~No keyword density or TF-IDF analysis~~ RESOLVED**
The `analyze_seo()` method now performs keyword density analysis on every page. It normalizes body text (lowercase, strips punctuation), filters out stop words and short tokens, and counts term frequency using `collections.Counter`. The output includes: `total_meaningful_words`, `top_keywords` (top 20 by frequency with `keyword`, `count`, and `density_pct`), and `prominent_keywords` (top 5 keywords checked for presence in `<title>` and `<h1>` elements). Results are nested under `seo.content.keyword_density` in each page's SEO data and aggregated in `seo-summary.json`. Note: TF-IDF across pages and user-supplied target keyword lists are not yet supported.

**21. ~~No sitemap.xml validation~~ RESOLVED**
When `--sitemap` is used, `validate_sitemap()` runs after crawling and cross-references sitemap URLs against actually scraped pages. The report identifies: pages missing from the sitemap (`scraped_not_in_sitemap`), sitemap URLs that weren't scraped (`sitemap_only`), and sitemap URLs that returned errors (`sitemap_errors`). Results are saved to `sitemap-validation.json` and included in `site-report.json` under `seo.sitemap_validation`. Note: `<lastmod>` date accuracy and sitemap protocol conformance are not yet validated.

**22. ~~No redirect chain analysis~~ RESOLVED**
The `fetch()` method now inspects `resp.history` after every request. When HTTP redirects occur, each hop's URL and status code are recorded alongside the original URL, final URL, and total hop count. After crawling, `redirect-chains.json` is written with all chains that have 1+ hops, capped at 100 entries. Multi-hop chains (>1 redirect) are highlighted separately. The report includes `total_redirects`, `multi_hop` count, and the full chain array for each redirect sequence.

**23. ~~No Core Web Vitals measurement~~ PARTIALLY RESOLVED**
In `--render-js` mode, LCP (Largest Contentful Paint) and CLS (Cumulative Layout Shift) are now measured via the browser's Performance API and reported per page under `seo.performance`. LCP is captured from `performance.getEntriesByType('largest-contentful-paint')` and CLS from `performance.getEntriesByType('layout-shift')`. Note: FID/INP cannot be measured without user interaction (they measure response to input events). These are lab metrics from a single load, not field data from real users.

**24. ~~No accessibility audit~~ RESOLVED**
A standalone `scripts/a11y_analyzer.py` evaluates scraped pages against 10 WCAG 2.1 Level AA criteria: language attribute (3.1.1), document title (2.4.2), heading hierarchy (1.3.1), image alt text quality (1.1.1), form labels (1.3.1/4.1.2), link purpose (2.4.4), ARIA validity (4.1.2), keyboard focus indicators (2.4.7), skip navigation (2.4.1), landmark regions (1.3.6), table headers (1.3.1), and color contrast heuristics (1.4.3). Each page receives a score from 0–100 with a letter grade (A–F). Output is saved to `a11y/a11y-analysis.json`, `a11y/a11y-summary.json`, and `a11y/per-page/`. The analyzer can also run on a single HTML file via `--page`.

---

### Local Serving & Viewing

**25. ~~No live reload / file watching~~ RESOLVED**
The `--live-reload` flag on `serve_site.py` starts a background file watcher that monitors `.html`, `.css`, and `.js` files for changes. When a change is detected, a live-reload script injected into HTML responses triggers an automatic browser refresh. The script uses a polling-based approach (no WebSocket dependency) — injected before `</body>` in every served HTML page. The watcher thread runs at 1-second intervals.

**26. ~~No HTTPS support on the local server~~ RESOLVED**
The `--https` flag on `serve_site.py` generates a self-signed TLS certificate (via `openssl`) and wraps the server socket with SSL. The certificate is stored as `.server-cert.pem` and `.server-key.pem` in the site directory and reused on subsequent runs. The browser will show a security warning (expected for self-signed certs), but secure-context APIs (service workers, Geolocation, Clipboard, etc.) will function correctly. Falls back to HTTP if `openssl` is not available.

**27. ~~No URL rewriting for internal page-to-page links~~ RESOLVED**
After all pages are scraped, a second pass (`rewrite_page_links()`) rewrites all `<a href>` links between scraped pages to use relative local paths via `os.path.relpath()`. The `page_url_map` dict maps URL variants (with/without trailing slash) to local paths. Fragments are preserved, query params stripped for lookup, and self-links reduced to `#fragment`. Multi-page sites are now fully navigable when served locally.

**28. No diff/comparison view when scraping the same site again later**
There is no mechanism to compare two scrapes of the same site taken at different times. Re-running the scraper on the same URL overwrites the output directory (or creates a new one). There is no snapshot versioning, no HTML diff generation, no asset change detection, and no side-by-side comparison view. If you want to track how a site's design or SEO changes over time, you must manually manage output directories and diff the files yourself.

---

### Output & Integration

**29. No HTML-to-React/Vue/Svelte component conversion**
The scraper outputs raw HTML files with rewritten asset paths. It does not convert the HTML into framework-specific component files. There is no JSX generation, no Vue SFC (`.vue`) generation, no Svelte component generation, and no component boundary detection (identifying which parts of the page should be separate components). Claude can manually read the HTML and write components from it, but there is no automated conversion pipeline.

**30. ~~No design token extraction~~ RESOLVED**
A standalone `scripts/extract_tokens.py` reads all CSS sources (external stylesheets, inline `<style>` blocks, and `style=""` attributes) from scraped output and produces structured design tokens in three formats: CSS custom properties (`:root { --color-primary: ... }`), W3C Design Tokens JSON (draft spec), and Tailwind config (`theme.extend` snippet). The extractor classifies colors into primary/neutral/accent groups by saturation and usage frequency, detects typography scales (font families + sorted font sizes), identifies spacing systems (base-4, base-8, or custom), catalogs border radii, box shadows, and media query breakpoints. Output is saved to `design-tokens/tokens.css`, `design-tokens/tokens-w3c.json`, `design-tokens/tailwind.config.js`, and `design-tokens/tokens-raw.json`.

**31. No Figma-compatible output**
There is no export to Figma's REST API, no `.fig` file generation, and no Figma plugin integration. The scraped design information (colors, fonts, layout) exists only in `site-report.json` and the raw CSS files.

**32. No screenshot comparison (before/after or original vs. scraped)**
The Playwright `screenshot` and `responsive` subcommands take screenshots of the locally served scraped content. However, there is no automated workflow to also screenshot the original live site and compare the two. There is no pixel-diff tool, no visual regression report, and no side-by-side comparison output. You cannot programmatically verify whether the local copy visually matches the original.

**33. ~~No export to Tailwind config from extracted design patterns~~ RESOLVED**
Resolved as part of gap #30 (design token extraction). Running `extract_tokens.py --format tailwind` (or `--format all`) produces a `design-tokens/tailwind.config.js` file containing a `theme.extend` snippet with colors, font families, font sizes, spacing, border radii, box shadows, and breakpoints populated from the scraped site's CSS.

**34. ~~No YAML output option on the main scraper~~ RESOLVED**
The main scraper now automatically writes `site-report.yaml` alongside `site-report.json` when the `pyyaml` package is installed. No additional flags needed — if `import yaml` succeeds, the YAML version is generated. Per-page SEO data remains JSON-only (the `seo_analyzer.py --format yaml` path covers that use case).

## Project Structure

```
web-scraper-seo/
├── SKILL.md                          # Skill instructions (what Claude reads)
├── README.md                         # This file
├── scripts/
│   ├── scrape_site.py                # Core scraping engine (~1500 lines)
│   │   └── WebScraper class
│   │       ├── fetch()               # HTTP with retries, SSL fallback
│   │       ├── check_robots_txt()    # robots.txt parsing
│   │       ├── _init_playwright()    # Lazy Playwright browser init (Gap #1)
│   │       ├── _fetch_with_playwright()  # JS-rendered page fetch (Gap #1)
│   │       ├── _close_playwright()   # Playwright cleanup (Gap #1)
│   │       ├── fetch_sitemap_urls()  # sitemap.xml/index parsing (Gap #3)
│   │       ├── save_checkpoint()     # Write crawl state to disk (Gap #7)
│   │       ├── load_checkpoint()     # Restore crawl state (Gap #7)
│   │       ├── clear_checkpoint()    # Remove checkpoint on success (Gap #7)
│   │       ├── record_link_source()  # Track page→URL link for broken detection (Gap #18)
│   │       ├── generate_broken_links_report()  # Cross-reference errors with sources (Gap #18)
│   │       ├── extract_nav_links()   # Navigation-scoped link discovery + link tracking
│   │       ├── extract_asset_urls()  # CSS/JS/image/font URL extraction from HTML
│   │       ├── extract_css_assets()  # @import and url() extraction from CSS
│   │       ├── download_asset()      # Asset downloading with dedup and size limits
│   │       ├── rewrite_html()        # Local path rewriting in HTML (assets + inline styles)
│   │       ├── _rewrite_css_urls()   # Parameterized CSS url() rewriting (shared helper)
│   │       ├── rewrite_css()         # External CSS url() rewriting (delegates to helper)
│   │       ├── rewrite_page_links()  # Second-pass <a href> rewriting for inter-page links
│   │       ├── scrape_page()         # Single page scrape (HTTP or Playwright)
│   │       ├── analyze_seo()         # Per-page SEO extraction + link source tracking
│   │       ├── _extract_css_design_patterns() # Shared color/font/layout extraction
│   │       ├── extract_design_notes()# Design system from CSS + inline styles
│   │       ├── generate_seo_summary()# Cross-page SEO aggregation
│   │       ├── build_sitemap_tree()  # Hierarchical sitemap from flat page list
│   │       └── crawl()              # Main BFS loop + sitemap + checkpoint + broken links
│   ├── serve_site.py                 # Local HTTP server (215 lines)
│   │   └── ScrapedSiteHandler        # Custom handler with path mapping + CORS
│   ├── playwright_view.py            # Playwright integration (285 lines)
│   │   ├── screenshot                # Single screenshot at specified viewport
│   │   ├── responsive                # Multi-viewport (desktop/tablet/mobile)
│   │   ├── render                    # JS-rendered HTML extraction
│   │   └── js-crawl                  # Full JS-rendered crawl with depth
│   ├── seo_analyzer.py               # Deep SEO analysis (425 lines)
│   │   ├── analyze_heading_hierarchy()
│   │   ├── analyze_meta_tags()
│   │   ├── analyze_links()
│   │   ├── analyze_images()
│   │   ├── analyze_technical()
│   │   ├── generate_seo_score()      # 0-100 scoring with letter grades
│   │   └── analyze_site()            # Full site analysis orchestrator
│   ├── extract_tokens.py             # Design token extractor (CSS/JSON/Tailwind)
│   │   ├── extract_colors()          # Color classification (primary/neutral/accent)
│   │   ├── extract_typography()      # Font families + size scale
│   │   ├── extract_spacing()         # Spacing system detection (base-4/8)
│   │   ├── generate_css_tokens()     # CSS custom properties output
│   │   ├── generate_w3c_tokens()     # W3C Design Tokens JSON output
│   │   ├── generate_tailwind_config()# Tailwind theme.extend output
│   │   └── main()                    # CLI entry point
│   └── a11y_analyzer.py              # Accessibility audit (~500 lines)
│       ├── check_language_attribute()     # WCAG 3.1.1
│       ├── check_heading_hierarchy()      # WCAG 1.3.1
│       ├── check_image_alt_text()         # WCAG 1.1.1
│       ├── check_form_labels()            # WCAG 1.3.1, 4.1.2
│       ├── check_link_text()              # WCAG 2.4.4
│       ├── check_aria_validity()          # WCAG 4.1.2
│       ├── check_keyboard_focus()         # WCAG 2.4.7
│       ├── check_skip_navigation()        # WCAG 2.4.1
│       ├── check_landmark_regions()       # WCAG 1.3.6
│       ├── check_table_accessibility()    # WCAG 1.3.1
│       ├── check_color_contrast_hints()   # WCAG 1.4.3
│       ├── generate_a11y_score()          # 0-100 scoring with letter grades
│       └── analyze_site()                 # Full site analysis orchestrator
├── agents/
│   └── web-analyzer.md               # Companion subagent for deep design analysis
├── references/
│   ├── seo-checklist.md              # SEO elements reference for mockup generation
│   └── site-report-schema.md         # Full site-report.json schema with examples
└── evals/
    └── evals.json                    # Test cases for skill validation
```

## License

This skill is provided for learning, development reference, and brainstorming purposes. Always respect the terms of service and copyright of any website you scrape. Downloaded content is intended for local development use only.
