
<h1 align="center">
  <img alt="ShikanokoBail banner" src="https://i.pinimg.com/736x/0c/ff/62/0cff624a04a81495f4b8e69bcedd34aa.jpg" width="100%"/>
</h1>

<div align="center">

[![npm](https://img.shields.io/npm/v/wsper-js?label=npm)](https://www.npmjs.com/package/wsper-js)
![TypeScript](https://img.shields.io/badge/TypeScript-5.8-blue)
![Node](https://img.shields.io/badge/Node-%3E%3D18.0.0-green)
![Tests](https://img.shields.io/badge/tests-383%20passed-brightgreen)
![License](https://img.shields.io/badge/license-GPL--3.0--or--later-blue)
</div>

# wsper-js
`wsper-js` is a TypeScript-first scraper toolkit for building reusable, typed, and testable scrapers across multiple public platforms. It provides platform-specific scraper classes, shared HTTP and queue primitives, safe downloader utilities, parser helpers, media utilities, and runnable examples for controlled and ethical data access.

The package is ESM-only and targets Node.js 18 or newer.

## Features

- Modular scraper architecture under `src/scrapers/`.
- TypeScript strict mode with exported response, option, and scraper data types.
- Standard `WsperResponse<T>` shape for scraper results.
- Shared HTTP client with timeouts, bounded retries, redirects, URL safety checks, browser-compatible public headers, and optional request queue pacing.
- Unit-tested scraper behavior and parser behavior.
- `examples/alllexamp.ts` runner for direct scraper demos and subprocess example checks.
- Local mock server support for testing WordPress-style endpoints, AI tools, and Fandom API responses without depending on live external services.
- Public API integrations where available, including LRCLIB, Wallhaven, Fandom MediaWiki, Spotify Web API, BMKG, npm registry, and BiliBili APIs.
- Cookie or credential support for platforms that legitimately require authenticated sessions, including Instagram, Twitter/X, Threads, Pinterest, BiliBili, and Spotify credentials.
- Additional modules for downloads, Brat image/GIF/video generation, and analytics chart image generation.

## Installation

```bash
npm install wsper-js
```

```bash
pnpm add wsper-js
```

```bash
yarn add wsper-js
```

Some optional capabilities need external binaries:

| Feature | External tool |
| --- | --- |
| YouTube and Spotify media enrichment/download helpers | `yt-dlp` |
| Video export and GIF/MP4 conversion | `ffmpeg` in `PATH` or configured path |

## Quick Start

```ts
import { LyricsScraper } from "wsper-js";

const lyrics = new LyricsScraper();
const result = await lyrics.search("after hours the weeknd");

if (!result.ok) {
  throw new Error(`${result.error?.code}: ${result.error?.message}`);
}

console.log(result.statusCode);
console.log(result.data?.title);
console.log(result.data?.lyrics);
```

You can also use `WsperScraper` for the aggregate social/media scraper entrypoint:

```ts
import { WsperScraper } from "wsper-js";

const wsper = new WsperScraper({
  queue: { concurrency: 1, minDelayMs: 500, maxDelayMs: 1500 },
});

const track = await wsper.spotify.search("never gonna give you up", { limit: 3 });
const video = await wsper.youtube.getVideo("dQw4w9WgXcQ");

console.log(track.ok, video.ok);
```

`WsperScraper` currently exposes `spotify`, `twitter`, `threads`, `instagram`, `pinterest`, `youtube`, and `cai`. Other scrapers are exported as named classes.

## Response Shape

Scraper methods return `WsperResponse<TData>`:

```ts
export interface WsperResponse<TData, TMeta extends WsperResponseMeta = WsperResponseMeta> {
  ok: boolean;
  statusCode: number;
  data: TData | null;
  error: {
    code: string;
    message: string;
    details?: Record<string, unknown>;
  } | null;
  meta: TMeta;
}

export interface WsperResponseMeta {
  statusCode: number;
  sourceUrl: string;
  fetchedAt: string;
  durationMs: number;
}
```

Recommended handling:

```ts
const response = await scraper.search("query");

if (response.ok && response.data !== null) {
  console.log(response.data);
} else {
  console.error(response.statusCode, response.error?.code, response.error?.message);
}
```

## Usage Examples

The examples below use public exports from `src/index.ts` and representative response fields from implementation and tests.

### Public-Link Examples

Some scrapers need live public targets. The bundled examples avoid dummy URLs:

```bash
npx tsx examples/gdrive.example.ts
npx tsx examples/foreign-news.example.ts
npx tsx examples/pubgmobile.example.ts
npx tsx examples/soundcloud.example.ts
```

`GDriveScraper` defaults to a verified public Google Drive PDF in the example. `ForeignNewsScraper` uses the BBC World RSS feed. `PubgMobileScraper` uses the official server-rendered announcement list because the public `news.shtml` page is a client-rendered shell. Scrapers that commonly require a valid session or cookie, such as TeraBox, Pixiv, and Xiaohongshu, require an explicit URL/ID in their examples instead of using placeholder links.

When a live endpoint returns HTTP 200 but no parseable data, these scrapers return `ok: false` with a parser error code instead of returning `ok: true` with empty data.

### LyricsScraper

Source: LRCLIB JSON API at `https://lrclib.net/api/search`.

```ts
import { LyricsScraper } from "wsper-js";

const scraper = new LyricsScraper();
const result = await scraper.search("after hours the weeknd");

console.log(result.data);
```

Representative output:

```json
{
  "title": "After Hours",
  "lyrics": "Oh, baby, where are you now?",
  "link": "https://lrclib.net/api/get/98765"
}
```

### WallpaperScraper

Source: Wallhaven search API at `https://wallhaven.cc/api/v1/search`.

```ts
import { WallpaperScraper } from "wsper-js";

const scraper = new WallpaperScraper();
const result = await scraper.search("cyberpunk");

console.log(result.data?.total);
console.log(result.data?.results[0]);
```

Representative output:

```json
{
  "total": 1,
  "results": [
    {
      "title": "Wallpaper sky123",
      "resolution": "1920x1080",
      "image": "https://w.wallhaven.cc/full/sky123.jpg",
      "page": "https://wallhaven.cc/w/sky123"
    }
  ]
}
```

### WwCharScraper

Source: Wuthering Waves Fandom MediaWiki parse API.

```ts
import { WwCharScraper } from "wsper-js";

const scraper = new WwCharScraper();
const result = await scraper.getCharacter("Jin Hsi");

console.log(result.data);
```

Representative output:

```json
{
  "title": "Jiyan",
  "slug": "Jin_Hsi",
  "url": "https://wutheringwaves.fandom.com/wiki/Jin_Hsi",
  "bio": "Bio karakter.",
  "profile": {},
  "images": []
}
```

### HokInfoScraper

Source: Honor of Kings Fandom MediaWiki parse API.

```ts
import { HokInfoScraper } from "wsper-js";

const scraper = new HokInfoScraper();
const result = await scraper.getCharacter("Angela");

console.log(result.data?.profile);
```

Representative output:

```json
{
  "title": "Sun Wukong",
  "image": null,
  "profile": {
    "Role": "Fighter"
  },
  "bio": "Bio singkat karakter.",
  "skills": [],
  "lore": null,
  "url": "https://honor-of-kings.fandom.com/wiki/Sun%20Wukong"
}
```

### CapCutScraper

Source: WordPress-style resolver endpoint, defaulting to `https://capdownloader.com/wp-json/aio-dl/video-data/`.

```ts
import { CapCutScraper } from "wsper-js";

const scraper = new CapCutScraper();
const result = await scraper.download("https://www.capcut.com/t/Zs82gHj1a/");

console.log(result.data);
```

Representative output:

```json
{
  "videoUrl": "https://cdn.example/video.mp4"
}
```

### ImgUpscalerScraper

Source: `https://get1.imglarger.com` upload and status endpoints.

```ts
import { readFile } from "node:fs/promises";
import { ImgUpscalerScraper } from "wsper-js";

const image = await readFile("./photo.jpg");
const scraper = new ImgUpscalerScraper();
const result = await scraper.upscaleBuffer(image, "photo.jpg", 4);

console.log(result.data);
```

Representative output:

```json
{
  "originalPath": null,
  "outputPath": null,
  "resultUrl": "https://cdn.test/out.jpg",
  "scale": 4
}
```

### PhotoAiScraper

Source: `https://photoai.imglarger.com`.

```ts
import { readFile } from "node:fs/promises";
import { PhotoAiScraper } from "wsper-js";

const image = await readFile("./portrait.jpg");
const scraper = new PhotoAiScraper();

const upload = await scraper.uploadBuffer(image, "portrait.jpg");
if (upload.ok && upload.data !== null) {
  const status = await scraper.checkStatus(upload.data.code);
  console.log(status.data);
}
```

Representative output:

```json
{
  "status": "success",
  "downloadUrl": "https://cdn.test/out.jpg",
  "raw": {
    "status": "success",
    "downloadUrl": "https://cdn.test/out.jpg"
  }
}
```

### FaceswapScraper

Source: `https://api.lovefaceswap.com`.

```ts
import { readFile } from "node:fs/promises";
import { FaceswapScraper } from "wsper-js";

const [source, target] = await Promise.all([
  readFile("./source-face.jpg"),
  readFile("./target.jpg"),
]);

const scraper = new FaceswapScraper();
const result = await scraper.process(source, target);

console.log(result.data);
```

Representative output:

```json
{
  "job_id": "job-1",
  "image": "https://cdn.test/out.jpg"
}
```

### UpscalerScraper

Source: `https://aienhancer.ai`.

```ts
import { readFile } from "node:fs/promises";
import { UpscalerScraper } from "wsper-js";

const image = await readFile("./photo.jpg");
const scraper = new UpscalerScraper();
const result = await scraper.upscaleBuffer(image, "image/jpeg");

console.log(result.data);
```

Representative output:

```json
{
  "id": "task-1",
  "input": "https://cdn.test/in.jpg",
  "output": "https://cdn.test/out.jpg"
}
```

### StalkScraper

Source: npm registry API at `https://registry.npmjs.org`.

```ts
import { StalkScraper } from "wsper-js";

const scraper = new StalkScraper();
const result = await scraper.getNpmPackage("axios");

console.log(result.data?.title);
console.log(result.data?.install);
```

Representative output:

```json
{
  "title": "@scope/pkg",
  "language": "",
  "publish": "2026-05-01T00:00:00.000Z",
  "readme": "",
  "explore": "https://www.npmjs.com/package/@scope/pkg",
  "dependencies": "0",
  "dependents": "0",
  "version_count": "1",
  "keywords": [],
  "install": "npm install @scope/pkg",
  "info": [],
  "collaborator": []
}
```

### MediafireScraper

Source: Mediafire public file page HTML.

```ts
import { MediafireScraper } from "wsper-js";

const scraper = new MediafireScraper();
const result = await scraper.getLink(
  "https://www.mediafire.com/file/ipnyzofjcwri357/test-10mb.bin/file",
);

console.log(result.data);
```

Representative output:

```json
{
  "downloadUrl": "https://download.example/myfile.zip",
  "fileName": "My_File.zip",
  "fileSize": "50 MB",
  "fileType": "ZIP"
}
```

### CaiScraper

Source: Character.AI WebSocket & API integrations.

```ts
import { CaiScraper } from "wsper-js";

const scraper = new CaiScraper({
  credentials: {
    bearerToken: "YOUR_CHARACTER_AI_TOKEN",
  },
});

// 1. Search for a character
const search = await scraper.searchCharacters("Mario");
console.log(search.data?.[0]);

// 2. Chat with a character (optionally using a custom voice ID)
const chat = await scraper.chat({
  characterId: "rGKdvZewGUZEJFEQPEBMS5JLQOTOrxi-8ByLFsGmgQM",
  message: "Hello Mario!",
  voiceId: "4fdd6bc1-c659-4587-b462-53f569b39078",
});
console.log(chat.data?.text);
console.log(chat.data?.audioUrl); // Generated TTS audio URL

// Chat sessions stay connected per character and are reused by later chat()
// calls. Disconnect them when your app is done with the character.
await scraper.disconnectCharacterSession("rGKdvZewGUZEJFEQPEBMS5JLQOTOrxi-8ByLFsGmgQM");
// Or:
await scraper.disconnectAllCharacterSessions();
```

Representative output:

```json
{
  "text": "It's-a me, Mario!",
  "audioUrl": "https://storage.googleapis.com/.../voice.mp3",
  "raw": { ... }
}
```

## Available Scrapers

| Scraper | Purpose | Source/API | Auth/Cookie required? | Example file | Notes |
| --- | --- | --- | --- | --- | --- |
| `AlkitabScraper` | Bible verse search | `alkitab.me` | No | `examples/alkitab.example.ts` | `search(query)` |
| `AnimeQuoteScraper` | Random anime quote | `otakotaku.com` | No | `examples/anime-quote.example.ts` | `getRandom()` |
| `AnimeRandomScraper` | Random anime character image | GitHub raw anime dataset | No | `examples/anime-random.example.ts` | `getImage(character)`, `random()` |
| `BiliBiliScraper` | BiliBili search and video info | `api.bilibili.com` | Optional cookie | `examples/bilibili.example.ts` | Cookie may unlock authenticated stream access |
| `BMKGScraper` | Indonesian earthquake and weather feeds | `data.bmkg.go.id`, `nowcasting.bmkg.go.id` | No | `examples/bmkg.example.ts` | Autogempa, gempa dirasakan, nowcasting, forecast |
| `CapCutScraper` | Resolve CapCut template video URL | `capdownloader.com/wp-json/aio-dl/video-data/` | No | `examples/capcut.example.ts` | Mocked in example runner |
| `CaiScraper` | Character.AI search, chat, and voice details | `neo.character.ai` API & WebSockets | Yes (Token required) | `examples/cai.example.ts` | `searchCharacters`, `chat`, `getVoice` |
| `CuacaScraper` | Indonesian weather by location/coordinate | BMKG weather APIs | Optional API key for warnings | `examples/cuaca.example.ts` | Reads optional `BMKG_WARNING_API_KEY` in example |
| `DrakorScraper` | Korean drama search/list/detail | `drakorkita30.kita.baby` | No | `examples/drakor.example.ts` | `search`, `detail`, `ongoing`, `getAll` |
| `DramaboxScraper` | Dramabox search | `dramabox.com` | No | `examples/dramabox.example.ts` | `search(query)` |
| `FaceswapScraper` | Face-swap image processing | `api.lovefaceswap.com` | No | `examples/faceswap.example.ts` | Mocked in example runner |
| `HokInfoScraper` | Honor of Kings character info | Fandom MediaWiki parse API | No | `examples/hok-info.example.ts` | Uses `api.php?action=parse` |
| `HtmlToJpgScraper` | HTML file to JPG conversion | `api.freeconvert.com` | No credential in code | `examples/html-to-jpg.example.ts` | File-based conversion; skipped in runner without fixture |
| `IkiruMangaScraper` | Manga search | `02.ikiru.wtf` | No | `examples/ikiru-manga.example.ts` | Mock server fallback available |
| `ImageScraper` | Safebooru image search | `safebooru.org` | No | `examples/image.example.ts` | Current site type: `safebooru` |
| `ImgUpscalerScraper` | Image upscaling | `get1.imglarger.com` | No | `examples/img-upscaler.example.ts` | Mocked in example runner |
| `InstagramScraper` | Profile, feed, post, download | `instagram.com` web/API endpoints | Internal defaults; custom cookie supported | `examples/instagram.example.ts` | Use only legitimate session cookies |
| `KomikindoScraper` | Manga search/detail | `komikindo.ch` | No | `examples/komikindo.example.ts` | `search`, `getDetail` |
| `LyricsScraper` | Lyrics search | `lrclib.net` JSON API | No | `examples/lyrics.example.ts` | Replaced blocked HTML scraping with API integration |
| `MConverterScraper` | File conversion helpers | `mconverter.eu` | No | `examples/mconverter.example.ts` | `getTargets`, `convert`, `convertBuffer` |
| `McAddonScraper` | Minecraft addon search/detail | `mmcreviews.com` | No | `examples/mcaddon.example.ts` | `search`, `getDetail`, `getAddon` |
| `MediafireScraper` | Resolve Mediafire download link | Mediafire public HTML page | No | `examples/mediafire.example.ts` | Default example uses active 10MB test file |
| `ModAndroidScraper` | Android APK/mod search aggregations | `an1.com`, `modyolo.com`, `aptoide.com`, `uptodown.com` | No | `examples/mod-android.example.ts` | `android1`, `modyolo`, `aptoide`, `uptodown`, `searchAll` |
| `OcrScraper` | OCR image scan | `newocr.com` | No | `examples/ocr.example.ts` | File/buffer based |
| `PhotoAiScraper` | Photo AI upload/status | `photoai.imglarger.com` | No | `examples/photo-ai.example.ts` | Mocked in example runner |
| `PinterestScraper` | Pin search, detail, download | `pinterest.com` | Internal defaults; custom cookie supported | `examples/pinterest.example.ts` | Supports `credentials` option |
| `PlayStoreScraper` | Google Play app search | `play.google.com` | No | `examples/playstore.example.ts` | `search(query, limit)` |
| `ResepScraper` | Recipe search | `cookpad.com` | No | `examples/resep.example.ts` | Returns recipe items |
| `SakuraNovelScraper` | Novel search/detail/chapter | `sakuranovel.id` | No | `examples/sakura-novel.example.ts` | Mock server fallback available |
| `SpotifyScraper` | Spotify track, album, playlist, search, downloads | Spotify Web API and Accounts API | Internal defaults; custom client credentials supported | `examples/spotify.example.ts` | Partial custom credentials are rejected |
| `StalkScraper` | npm package metadata lookup | `registry.npmjs.org` | No | `examples/stalk.example.ts` | Default example query is `axios` |
| `ThreadsScraper` | Threads profile, post, search, download | `threads.net` web/API endpoints | Internal defaults; custom cookie supported | `examples/threads.example.ts` | Use only legitimate session cookies |
| `TopAnimeScraper` | MyAnimeList top anime list | `myanimelist.net` | No | `examples/top-anime.example.ts` | `getTopAnime(limit)` |
| `TwitterScraper` | Tweet, profile, search, timeline, downloads | `x.com/i/api/graphql` | Cookie/CSRF commonly required | `examples/twitter.example.ts` | Supports `credentials` option |
| `UguuScraper` | Temporary file upload | `uguu.se/upload` | No | `examples/uguu.example.ts` | `upload(buffer, filename)` |
| `UpscalerScraper` | Image enhancement | `aienhancer.ai` | No | `examples/upscaler.example.ts` | Rejects remote URL string input |
| `VideyScraper` | Video upload | `videy.co/api/upload` | No | `examples/videy.example.ts` | `upload`, `uploadBuffer` |
| `WallpaperScraper` | Wallpaper search | `wallhaven.cc/api/v1/search` | No | `examples/wallpaper.example.ts` | Replaced blocked HTML scraping with API integration |
| `Webp2Mp4Scraper` | WebP to MP4/PNG conversion | `ezgif.com` | No | `examples/webp2mp4.example.ts` | `toMp4`, `toPng` |
| `WwCharScraper` | Wuthering Waves character info | Fandom MediaWiki parse API | No | `examples/ww-char.example.ts` | Uses `api.php?action=parse` |
| `YouTubeScraper` | YouTube metadata, search, playlist, channel, downloads | `yt-dlp`, `play-dl`, YouTube pages | No cookie option exposed in current scraper options | `examples/youtube.example.ts` | Media download features need external tools |

## What's New

The latest scraper reliability pass fixed 12 failing scraper functions and their corresponding tests.

- `LyricsScraper` now uses LRCLIB's public JSON API instead of a Cloudflare-protected Musixmatch HTML page.
- `WallpaperScraper` now uses Wallhaven's public search API instead of a Cloudflare-protected Wallpaperflare HTML page.
- `WwCharScraper` and `HokInfoScraper` now use Fandom's MediaWiki parse API (`api.php?action=parse`) and prepend a synthetic `#firstHeading` node before running the existing parsers.
- `examples/mock-server.ts` now covers WordPress resolver routes, AI tool polling/upload routes, and Fandom API responses.
- `examples/alllexamp.ts` starts and stops the mock server during the runner lifecycle.
- `WSPER_MOCK_BASE_URL` is injected into spawned example subprocesses so individual examples can use the local mock endpoint automatically.
- Default example inputs were updated: `StalkScraper` uses `axios`, `MediafireScraper` uses an active 10MB test file URL, and `HokInfoScraper` uses `Angela`.
- Verification recorded in `walkthrough.md`: 82 test files passed, 314 tests passed, and the examples runner reported 83 OK, 0 FAIL, 2 SKIP.

## Mock Server and Testing

`examples/mock-server.ts` is a local HTTP server used by the examples runner to provide deterministic responses for endpoints that are rate-limited, depend on third-party availability, or are inconvenient to call during automated checks.

Currently mocked routes include:

| Area | Routes |
| --- | --- |
| CapCut | `/wp-json/aio-dl/video-data/` |
| ImgUpscaler | `/api/UpscalerNew/UploadNew`, `/api/UpscalerNew/CheckStatusNew` |
| PhotoAi | `/api/PhoAi/Upload`, `/api/PhoAi/CheckStatus` |
| Faceswap | `/api/face-swap/create-poll`, `/api/common/get` |
| Upscaler | `/api/v1/r/image-enhance/create`, `/api/v1/r/image-enhance/result` |
| Fandom Wiki | `/api.php` |
| WordPress search fixtures | `/wp-admin/admin-ajax.php` |

`examples/alllexamp.ts` starts the mock server, sets `process.env.WSPER_MOCK_BASE_URL`, routes direct scraper demos to the mock server with `http: { allowPrivateNetwork: true }` where needed, runs individual example files as subprocesses, then closes the mock server.

This keeps example validation safer and more repeatable. It avoids making every test depend on live third-party services, Cloudflare-protected HTML pages, or rate-limited AI tool endpoints.

## Running Examples

Run all direct scraper demos and individual example files:

```bash
npx tsx examples/alllexamp.ts
```

The runner writes scraper JSON results to `downloads/output/scrapers/` and subprocess logs to `downloads/output/examples/`.

Summary fields:

| Field | Meaning |
| --- | --- |
| `OK` | The scraper/example returned a successful result. |
| `FAIL` | The scraper/example returned a failed response or subprocess exit. Auth-required scrapers may fail without valid credentials. |
| `SKIP` | The runner intentionally skipped an entry, usually because a local fixture is unavailable. |

Recorded walkthrough result:

```txt
OK   : 83
FAIL : 0
SKIP : 2
Total: 85
```

Individual examples can also be run directly:

```bash
npx tsx examples/lyrics.example.ts "after hours the weeknd"
npx tsx examples/wallpaper.example.ts cyberpunk
npx tsx examples/stalk.example.ts axios
npx tsx examples/mediafire.example.ts "https://www.mediafire.com/file/ipnyzofjcwri357/test-10mb.bin/file"
npx tsx examples/upscaler.example.ts testassets/photo.jpg
```

For scrapers that support cookies or credentials, use only accounts and sessions you are authorized to access. Do not hardcode real cookies, tokens, client secrets, or API keys in source files.

## Running Tests and Validation

Package scripts from `package.json`:

| Command | Purpose |
| --- | --- |
| `npm run test` | Run all Vitest tests once. |
| `npm run test:watch` | Run Vitest in watch mode. |
| `npm run typecheck` | Run TypeScript with `--noEmit`. |
| `npm run build` | Build production ESM output through `script/build.mjs`. |
| `npm run build:dev` | Build development output. |
| `npm run build:prod` | Build production output. |
| `npm run build:bytecode` | Build bytecode output with `script/build-bytecode.mjs`. |
| `npm run build:all` | Build production output and bytecode. |
| `npm run test:instagram` | Run Instagram tests only. |
| `npm run test:spotify` | Run Spotify tests only. |
| `npm run test:youtube` | Run YouTube tests only. |
| `npm run test:threads` | Run Threads tests only. |
| `npm run test:pinterest` | Run Pinterest tests only. |
| `npm run test:brat` | Run Brat tests only. |

Recommended validation before publishing or changing behavior:

```bash
npm run typecheck
npm run test
npm run build
npx tsx examples/alllexamp.ts
```

## Project Structure

```txt
src/
  index.ts                 Public package exports
  WsperScraper.ts          Aggregate scraper entrypoint
  core/
    credentials/           Credential normalization and platform headers
    error/                 WsperError, ValidationError, HttpError, ParseError, DownloadError, ScraperError
    http/                  HTTP client, retries, timeouts, safe URL validation
    parser/                Shared HTML and JSON parser helpers
    queue/                 Request pacing and concurrency control
  modules/
    brat/                  Brat image/GIF/video generator and converters
    chart/                 Analytics image generator
    download/              Safe downloader primitives
  scrapers/                Platform-specific scraper implementations
  types/                   Shared response, option, and common types
  utils/                   Sleep, URL, validation, browser-profile, and helper utilities
examples/
  alllexamp.ts             Full example runner
  mock-server.ts           Local deterministic mock server
  *.example.ts             Individual runnable examples
tests/
  */*.test.ts              Unit and parser tests
dist/                      Build output only; do not edit manually
```

## Environment Variables

These variables appear in the repository:

| Variable | Used by | Required? | Notes |
| --- | --- | --- | --- |
| `WSPER_MOCK_BASE_URL` | `examples/alllexamp.ts`, AI tool examples | No | Set by the all-examples runner for subprocesses. Points examples to the local mock server. |
| `BMKG_WARNING_API_KEY` | `examples/cuaca.example.ts` | No | Optional warning API key passed to `CuacaScraper({ warningApiKey })`. |
| `INSTAGRAM_COOKIE` | `examples/instagram.example.ts` comments | No | Optional example input for constructor credentials. Use `<YOUR_COOKIE_HERE>` style placeholders in docs and never commit real cookies. |
| `INSTAGRAM_CSRF_TOKEN` | `examples/instagram.example.ts` comments | No | Optional example input for constructor credentials. |
| `BILI_COOKIE` | `examples/bilibili.example.ts` | No | Optional BiliBili cookie for authenticated stream access. |
| `WSPER_COOKIE` | `tests/core/credentials.test.ts` | No | Test-only variable proving runtime credential resolution does not read env credentials automatically. |

Credential configuration is constructor-based. The library does not rely on `.env` files for runtime credentials.

```ts
import { WsperScraper } from "wsper-js";

const wsper = new WsperScraper({
  spotifyCredentials: {
    clientId: "your-client-id",
    clientSecret: "your-client-secret",
  },
  credentials: {
    cai: {
      bearerToken: "<YOUR_TOKEN_HERE>",
    },
    twitter: {
      cookie: "<YOUR_COOKIE_HERE>",
      csrfToken: "<YOUR_CSRF_TOKEN_HERE>",
    },
    instagram: {
      cookie: "<YOUR_COOKIE_HERE>",
      csrfToken: "<YOUR_CSRF_TOKEN_HERE>",
    },
  },
});
```

Spotify custom credentials must include both `clientId` and `clientSecret`. Partial custom Spotify credentials are rejected with `ValidationError`.

## Cloudflare, Rate Limits, and Reliability

Some websites protect HTML pages with Cloudflare, require active authenticated sessions, or apply strict rate limits. `wsper-js` favors public APIs when they are available and documented by implementation, such as LRCLIB, Wallhaven, Fandom MediaWiki, Spotify, BMKG, BiliBili, and npm registry endpoints.

The mock server exists to make tests and examples deterministic. It should be used for local validation of rate-limited, Cloudflare-protected, or external-service-dependent flows instead of repeatedly hitting live services.

This project does not provide CAPTCHA bypass logic, credential harvesting, session theft, anti-bot evasion, or rate-limit abuse. Use cookies only for accounts and sessions you legitimately control, respect platform terms, and keep request pacing conservative.

## Contributing

```bash
git clone <repository-url>
cd wsper
npm install
npm run typecheck
npm run test
```

When adding or changing a scraper:

1. Keep scraper-specific logic under `src/scrapers/<name>/`.
2. Export the scraper from `src/scrapers/<name>/index.ts` and `src/scrapers/index.ts`.
3. Return typed `WsperResponse<T>` results.
4. Keep HTTP, parsing, queueing, and file download responsibilities separated.
5. Add or update parser and scraper tests under `tests/`.
6. Add or update a runnable example under `examples/`.
7. Use the mock server for flows that should not depend on live third-party behavior in default tests.
8. Update this README when public API, usage, behavior, examples, or validation results change.

## Security

- Validate and normalize user-provided URLs before requesting them.
- Use only `http:` and `https:` unless a module explicitly supports something else.
- Keep SSRF protections enabled; private network requests require explicit `allowPrivateNetwork: true` and should be reserved for local mocks or trusted endpoints.
- Do not log secrets, cookies, client secrets, authorization headers, access tokens, refresh tokens, or raw credential objects.
- Do not commit credentials, cookies, tokens, private fixtures, or real session material.

## License

GPL-3.0-or-later. See `LICENSE` for the full license text.
