\n\n\n# Meta-Search Web Sources API\r\n\r\n- **Multi-engine aggregation**: 70+ major sites search sources across 10 categories\r\n- **Smart result merging**: Intelligent deduplication and ranking\r\n- **Weighted scoring**: Position-based algorithm with engine and category weights\r\n- **Category-based search**: Filter by academic, news, images, videos, etc.\r\n- **Autocomplete**: Query suggestions from 8 search engines\r\n\r\n## Search Site Sources\r\n\r\n### General Search\r\n\r\n| Engine | Description |\r\n| ------------- | ------------------------------------------------------------------- |\r\n| Google | The world's most popular web search engine |\r\n| Bing | Microsoft's web search engine for the general web |\r\n| Brave | Privacy-focused web search integrated with the Brave browser |\r\n| DuckDuckGo | Privacy-oriented search engine that does not track users |\r\n| Qwant | European privacy-focused web search engine |\r\n| Startpage | Search engine that proxies Google results with enhanced privacy |\r\n| Yahoo | Web portal providing search, email, and news |\r\n| Ask | Question-answering oriented web search engine |\r\n| Wikipedia | Free, community-edited online encyclopedia |\r\n| Wikidata | Structured, collaborative knowledge base from Wikimedia |\r\n| Wolfram Alpha | Computational knowledge engine for factual queries and calculations |\r\n\r\n### News\r\n\r\n| Engine | Description |\r\n| --------------- | ---------------------------------------------------------------------- |\r\n| DuckDuckGo News | Aggregated news search powered by DuckDuckGo |\r\n| Mojeek News | News search vertical from the independent Mojeek engine |\r\n| Bing News | News aggregation and search from Microsoft Bing |\r\n| Brave News | Privacy-focused news search from Brave Search |\r\n| Google News | Google's personalized news aggregation and search service |\r\n| Qwant News | News search and headlines from Qwant |\r\n| Yahoo News | News portal and search from Yahoo |\r\n| Tagesschau (DE) | German-language news search for Tagesschau, a major public broadcaster |\r\n\r\n### Videos\r\n\r\n| Engine | Description |\r\n| ----------------- | ---------------------------------------------------------------------------- |\r\n| Bing Videos | Video search vertical from Microsoft Bing |\r\n| Brave Videos | Video search vertical of Brave Search |\r\n| DuckDuckGo Videos | Video search results from DuckDuckGo |\r\n| Google Videos | Video-focused search results from Google |\r\n| Qwant Videos | Video search vertical from Qwant |\r\n| Bilibili | Chinese video sharing and streaming platform popular for animation and games |\r\n| Dailymotion | Global video-sharing and hosting platform |\r\n| Invidious | Alternative, privacy-friendly front-end for YouTube content |\r\n| Odysee | Decentralized, blockchain-based video hosting platform |\r\n| PeerTube | Federated, open source video hosting network |\r\n| Piped | Privacy-respecting alternative front-end for YouTube |\r\n| Rumble | Video sharing platform focusing on independent creators |\r\n| Vimeo | Ad-free video hosting platform used by filmmakers and businesses |\r\n| YouTube | The largest global video sharing and streaming platform |\r\n\r\n### Images\r\n\r\n| Engine | Description |\r\n| ----------------- | -------------------------------------------------------------- |\r\n| Bing Images | Image search engine by Microsoft Bing |\r\n| Brave Images | Image search vertical of Brave Search |\r\n| DuckDuckGo Images | Image search results from DuckDuckGo |\r\n| Google Images | Google's dedicated image search service |\r\n| Qwant Images | Image search vertical from Qwant |\r\n| DeviantArt | Online community and gallery for digital art and illustrations |\r\n| Flickr | Image and short video hosting platform for photographers |\r\n| Imgur | Image hosting and sharing site popular for memes and galleries |\r\n| Pinterest | Visual discovery and bookmarking platform centered on images |\r\n| Unsplash | Library of freely usable, high-resolution photos |\r\n| Wallhaven | Community-driven wallpaper and background image repository |\r\n| Wikimedia Commons | Image search across media stored on Wikimedia Commons |\r\n\r\n### Science & Academic\r\n\r\n| Engine | Description |\r\n| ------------------------ | ------------------------------------------------------------------------ |\r\n| arXiv | Repository of preprint research papers in physics, math, CS and more |\r\n| Crossref | Infrastructure service providing DOIs and metadata for scholarly content |\r\n| Google Scholar | Google's search engine for scholarly literature and citations |\r\n| Internet Archive Scholar | Internet Archive search for digitized scholarly works |\r\n| PubMed | Database of biomedical and life sciences literature from NCBI |\r\n| Semantic Scholar | AI-powered literature search and discovery tool for research papers |\r\n\r\n### IT & Development\r\n\r\n| Engine | Description |\r\n| --------------------- | --------------------------------------------------------------------- |\r\n| Alpine Linux Packages | Search index for Alpine Linux software packages |\r\n| Crates.io | Official package registry for Rust crates |\r\n| Docker Hub | Central repository of Docker container images |\r\n| Hex | Package manager and repository for the Erlang and Elixir ecosystem |\r\n| Hoogle | API search engine for Haskell libraries and functions |\r\n| Lib.rs | Alternative index and front-end for Rust crates |\r\n| MetaCPAN | Search engine for Perl modules hosted on CPAN |\r\n| npm | Package registry and manager for JavaScript and Node.js |\r\n| Packagist | Primary package repository for PHP Composer |\r\n| pkg.go.dev | Documentation and discovery service for Go modules |\r\n| pub.dev | Package repository for Dart and Flutter packages |\r\n| PyPI | Python Package Index, the main repository for Python packages |\r\n| RubyGems | Package manager and repository for Ruby gems |\r\n| Void Linux | Package search for the Void Linux distribution |\r\n| Ask Ubuntu | Q&A site focused on Ubuntu usage and troubleshooting |\r\n| Stack Overflow | Question-and-answer site for programmers and software development |\r\n| Super User | Q&A site for advanced computer users and system administration |\r\n| Bitbucket | Git-based source code hosting and collaboration platform by Atlassian |\r\n| Codeberg | Non-profit, open source Git hosting platform |\r\n| Gitea | Hosted instance of the lightweight Gitea Git service |\r\n| GitHub | Popular Git-based code hosting and collaboration platform |\r\n| GitLab | DevOps platform offering Git hosting, CI/CD, and project management |\r\n| SourceHut | Minimalist suite of tools for software development and mailing lists |\r\n| Arch Linux Wiki | Extensive documentation wiki for Arch Linux and related tools |\r\n| Gentoo | Documentation wiki for the Gentoo Linux distribution |\r\n| MDN | Mozilla Developer Network documentation for web technologies |\r\n| SearchCode | Search engine indexing source code across public repositories |\r\n\r\n### Files & Archives\r\n\r\n| Engine | Description |\r\n| ------------------------ | ----------------------------------------------------------------- |\r\n| Library of Congress | Catalog and digital collections from the U.S. Library of Congress |\r\n| Wikibooks | Wikimedia project hosting free textbooks and manuals |\r\n| Wikisource | Wikimedia digital library of source texts and documents |\r\n| Wikiversity | Wikimedia project for educational resources and courses |\r\n| Wikivoyage | Free, community-created travel guide from Wikimedia |\r\n| Google Play Movies | Google's store for purchasing and renting movies and TV series |\r\n| MediathekViewWeb (DE) | Web interface to German public TV media libraries |\r\n| INA (FR) | French National Audiovisual Institute archive search |\r\n| Wikimedia Commons Videos | Search for video media stored on Wikimedia Commons |\r\n| Wikimedia Commons Audio | Search for audio files stored on Wikimedia Commons |\r\n| APK Mirror | Repository for Android APK files outside official app stores |\r\n| Apple App Store | Official iOS and iPadOS app distribution platform |\r\n| F-Droid | Catalog of free and open source Android applications |\r\n| Google Play Apps | Official Android app store operated by Google |\r\n\r\n### Social Media\r\n\r\n| Engine | Description |\r\n| --------- | ----------------------------------------------------------------------- |\r\n| Pinterest | Visual discovery, bookmarking, and inspiration social platform |\r\n| Reddit | Social news aggregation and discussion website organized by communities |\r\n| 9GAG | Humor-focused social media site for sharing memes and short posts |\r\n| Lemmy | Post search across Lemmy, a federated link-aggregation platform |\r\n\r\n### Maps & Navigation\r\n\r\n| Engine | Description |\r\n| ------------- | ------------------------------------------------------ |\r\n| Apple Maps | Apple's mapping and navigation service for its devices |\r\n| OpenStreetMap | Collaborative, open data world map project |\r\n\r\n### Music\r\n\r\n| Engine | Description |\r\n| ------------- | ------------------------------------------------------------------ |\r\n| Genius | Platform for song lyrics, annotations, and music commentary |\r\n| Radio Browser | Directory and search engine for online radio stations |\r\n| Bandcamp | Music platform for independent artists to sell and stream releases |\r\n| Deezer | On-demand music streaming service |\r\n| Mixcloud | Streaming platform for DJ mixes, radio shows, and podcasts |\r\n| SoundCloud | Audio distribution and music sharing platform for creators |\r\n| YouTube | Video platform heavily used for streaming music and music videos |\r\n\r\n## API Usage\r\n\r\n### Basic Search\r\n\r\nSearch across all engines:\r\n\r\n```bash\r\ncurl \"http://localhost:3000?q=artificial+intelligence\"\r\n```\r\n\r\nSearch specific engines:\r\n\r\n```bash\r\ncurl \"http://localhost:3000?q=machine+learning&engines=google,duckduckgo,bing\"\r\n```\r\n\r\nSearch by category:\r\n\r\n```bash\r\ncurl \"http://localhost:3000?q=quantum+computing&categories=science\"\r\n```\r\n\r\n### Autocomplete\r\n\r\nGet suggestions from a single backend:\r\n\r\n```bash\r\ncurl \"http://localhost:3000/autocomplete?q=python&backend=google\"\r\n```\r\n\r\nGet suggestions from multiple backends:\r\n\r\n```bash\r\ncurl \"http://localhost:3000/autocomplete/multi?q=javascript&backends=google,duckduckgo,wikipedia\"\r\n```\r\n\r\nList available autocomplete backends:\r\n\r\n```bash\r\ncurl \"http://localhost:3000/autocomplete/backends\"\r\n```\r\n\r\n### API Documentation\r\n\r\nInteractive API documentation is available at:\r\n\r\n- Scalar UI: `http://localhost:3000/docs`\r\n- OpenAPI JSON: `http://localhost:3000/openapi.json`\r\n\r\n## Architecture\r\n\r\n### Technology Stack\r\n\r\n- **Runtime**: Bun (high-performance JavaScript runtime)\r\n- **Framework**: Hono (fast, lightweight web framework)\r\n- **HTTP Client**: grab-url (for all engine requests)\r\n- **HTML Parser**: linkedom (DOM parsing for scrapers)\r\n- **Language**: TypeScript\r\n\r\n### Ranking Algorithm\r\n\r\nResults are scored using SearXNG's proven algorithm:\r\n\r\n```\r\nscore = Σ (engine_weight × category_weight × occurrences / position)\r\n```\r\n\r\n**Variables:**\r\n\r\n- `engine_weight`: Configurable per-engine multiplier (default: 1.0)\r\n- `category_weight`: Category importance multiplier (e.g., academic: 1.3)\r\n- `occurrences`: Number of engines that found this result\r\n- `position`: Result position in original engine results\r\n\r\nCustomize engine importance in `app/lib/search.ts`:\r\nConfigure in `app/lib/category-registry.ts`:\r\n\r\n**Features:**\r\n\r\n- Deduplication by URL normalization\r\n- Multi-engine result merging\r\n- Category-based grouping\r\n- Priority levels (low, normal, high)\r\n\r\n### Project Structure\r\n\r\n```\r\napp/\r\n├── engines/ # Search engine implementations\r\n│ ├── general/ # Google, Bing, DuckDuckGo, etc.\r\n│ ├── academic/ # arXiv, Google Scholar, PubMed\r\n│ ├── images/ # Unsplash, Flickr, Imgur\r\n│ ├── videos/ # YouTube, Vimeo, Dailymotion\r\n│ ├── news/ # HackerNews, Google News, Bing News\r\n│ ├── it/ # GitHub, npm, PyPI, Stack Overflow\r\n│ ├── social/ # Reddit, Twitter, Medium\r\n│ ├── maps/ # OpenStreetMap, Apple Maps\r\n│ ├── shopping/ # eBay\r\n│ ├── torrents/ # 1337x, PirateBay, Nyaa\r\n│ └── specialized/ # Wikipedia, IMDb, Genius\r\n├── lib/ # Core search logic\r\n│ ├── search.ts # Main search orchestrator\r\n│ ├── result-container.ts # Result merging & ranking\r\n│ ├── category-registry.ts # Category management\r\n│ ├── engine.ts # Engine interface\r\n│ └── autocomplete.ts # Autocomplete backends\r\n├── routes/ # API routes\r\n│ ├── search.ts\r\n│ └── autocomplete.ts\r\n├── data/ # Static data\r\n│ └── engine-descriptions.ts\r\n└── index.ts # Main entry point\r\n```\r\n\r\n## Testing\r\n\r\n```bash\r\nbun run test # the suites CI gates on — no network\r\nbun run test:coverage # the same, with coverage (writes coverage/lcov.info)\r\nbun run test:live # adds the suites that call the real search engines\r\n```\r\n\r\n`bun run test` runs only deterministic suites: `test/sources-unit.test.ts`\r\nexercises every engine against mocked responses, and\r\n`src/search/__tests__/public-searxng.test.ts` covers the SearXNG client.\r\n\r\n`test/api.test.ts`, `test/search.test.ts`, `test/sources.test.ts`,\r\n`test/engine-health-suite.test.ts` and `test/autocomplete-ai.test.ts` really do\r\nquery the upstream engines, so whether they pass depends on third-party\r\navailability and on whether an engine feels like rate-limiting your IP. They\r\nare excluded unless `RUN_LIVE_TESTS=1` is set — which is what `test:live` does\r\n— and they are the right thing to run by hand when you change an engine, since\r\na mock only proves the parser still matches the fixture.\r\n\r\n`examples/autocomplete-engines.ts` (`bun run example:autocomplete`) prints what\r\neach autocomplete backend answers. It asserts nothing, so it is an example\r\nrather than a suite.\r\n","readmeFilename":"README.md"}
![](\"https://i.imgur.com/BmaDQeR.png\")
\n![\"Documentation\"](\"https://img.shields.io/badge/Docs-blue?logo=ReadTheDocs&logoColor=white\"){kind=icon}
![\"GitHub](\"https://img.shields.io/github/stars/OpenSourceAGI/qwksearch-research-agent\")
![\"NPM](\"https://img.shields.io/npm/dm/search-web-api.svg\")
\n ![\"npm](\"https://img.shields.io/npm/v/search-web-api.svg\")
\n ![\"NPM](\"https://img.shields.io/npm/dt/search-web-api.svg\")
\n ![\"TypeScript](\"https://img.shields.io/npm/types/search-web-api\")
![\"GitHub](\"https://img.shields.io/github/issues/OpenSourceAGI/qwksearch-research-agent?logo=github\")
![\"Open](\"https://img.shields.io/github/issues-pr/OpenSourceAGI/qwksearch-research-agent?logo=github&label=PRs\")
![\"Merged](\"https://img.shields.io/github/issues-pr-closed/OpenSourceAGI/qwksearch-research-agent?logo=github&label=PRs%20merged&color=8957e5\")
![\"GitHub](\"https://img.shields.io/github/discussions/OpenSourceAGI/qwksearch-research-agent\")
![\"GitHub](\"https://img.shields.io/github/last-commit/OpenSourceAGI/qwksearch-research-agent.svg\")
\n ![\"Open](\"https://developer.stackblitz.com/img/open_in_stackblitz.svg\")
\n ![\"Bun\"](\"https://img.shields.io/badge/Bun-14151A?logo=bun&logoColor=white\"){kind=icon}
![\"TypeScript\"](\"https://img.shields.io/badge/TypeScript-3178C6?logo=typescript&logoColor=white\"){kind=icon}
![\"Hono\"](\"https://img.shields.io/badge/Hono-E36002?logo=hono&logoColor=white\"){kind=icon}
![\"Vitest\"](\"https://img.shields.io/badge/Vitest-6E9F18?logo=vitest&logoColor=white\"){kind=icon}