{"_id":"@agixbt/elizascraper","_rev":"2-af9f248625b166918eb5075eb3a2b6e0","name":"@agixbt/elizascraper","dist-tags":{"latest":"1.0.1"},"versions":{"1.0.0":{"name":"@agixbt/elizascraper","version":"1.0.0","keywords":["blockchain","scraper","bera","cli","api"],"_id":"@agixbt/elizascraper@1.0.0","maintainers":[{"name":"agixbt","email":"foragixbt@proton.me"}],"bin":{"elizascraper":"dist/cli.js"},"dist":{"shasum":"fc73845a4029737100c8492adda92aa7ca278572","tarball":"https://registry.npmjs.org/@agixbt/elizascraper/-/elizascraper-1.0.0.tgz","fileCount":20,"integrity":"sha512-zTogHVkZ6WYg8HDmyPmAZ++rmkuG+hnTNb0zYeJ6ZbGfHz5qQ1CYeaTV59SOy3oVagdJlAWivKvKDON1IHoM5Q==","signatures":[{"sig":"MEQCIBJ1UMw3uscgZmtlwOHXjE0r1K+ly+ADyh0vwZsE1i5PAiBqDWA0xceuWHYTQSU41xx6cEQXx2vk3EzjkxdFOxS+eA==","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":2938306},"main":"dist/server.js","type":"module","types":"dist/server.d.ts","module":"dist/server.js","exports":{".":{"types":"./dist/server.d.ts","import":"./dist/server.js"},"./site":{"types":"./dist/lib/scraper.d.ts","import":"./dist/lib/scraper.js"},"./token":{"types":"./dist/lib/coingecko.d.ts","import":"./dist/lib/coingecko.js"},"./tweets":{"types":"./dist/lib/tweets.d.ts","import":"./dist/lib/tweets.js"},"./embeddings":{"types":"./dist/lib/embeddings.d.ts","import":"./dist/lib/embeddings.js"}},"gitHead":"0e1b52e5919b454adeb30acca933b2fc7920fdb0","scripts":{"cli":"bun run src/cli.ts","dev":"bun run --watch src/server.ts","build":"bun build ./src/server.ts ./src/cli.ts --outdir ./dist --target node --external pg --external @qdrant/js-client-rest --external playwright --external apify-client --format esm","postbuild":"tsc --declaration --emitDeclarationOnly --declarationDir dist","prepublishOnly":"bun run build"},"_npmUser":{"name":"agixbt","email":"foragixbt@proton.me"},"_npmVersion":"10.2.4","description":"Awesome Scraper for eliza, scrape docs, tweets, and tokens","directories":{},"_nodeVersion":"18.19.1","dependencies":{"v":"^0.3.0","pg":"^8.13.3","dotenv":"^16.4.7","elysia":"latest","openai":"^4.86.1","cheerio":"^1.0.0","commander":"^13.1.0","playwright":"^1.50.1","drizzle-orm":"^0.40.0","apify-client":"^2.11.1","@elysiajs/cors":"^1.2.0","@elysiajs/cron":"^1.2.0","@elysiajs/swagger":"^1.2.0","@qdrant/js-client-rest":"^1.13.0"},"_hasShrinkwrap":false,"devDependencies":{"@types/pg":"^8.11.11","bun-types":"latest","typescript":"^5.3.3"},"_npmOperationalInternal":{"tmp":"tmp/elizascraper_1.0.0_1741629183378_0.6769076336495103","host":"s3://npm-registry-packages-npm-production"}},"1.0.1":{"name":"@agixbt/elizascraper","version":"1.0.1","description":"Awesome Scraper for eliza, scrape docs, tweets, and tokens","type":"module","main":"dist/server.js","types":"dist/server.d.ts","bin":{"elizascraper":"dist/cli.js"},"scripts":{"dev":"bun run --watch src/server.ts","build":"bun build ./src/server.ts ./src/cli.ts --outdir ./dist --target node --external pg --external @qdrant/js-client-rest --external playwright --external apify-client --format esm","postbuild":"tsc --declaration --emitDeclarationOnly --declarationDir dist","cli":"bun run src/cli.ts","prepublishOnly":"bun run build"},"dependencies":{"@elysiajs/cors":"^1.2.0","@elysiajs/cron":"^1.2.0","@elysiajs/swagger":"^1.2.0","@qdrant/js-client-rest":"^1.13.0","apify-client":"^2.11.1","cheerio":"^1.0.0","commander":"^13.1.0","dotenv":"^16.4.7","drizzle-orm":"^0.40.0","elysia":"latest","openai":"^4.86.1","pg":"^8.13.3","playwright":"^1.50.1","v":"^0.3.0"},"devDependencies":{"@types/pg":"^8.11.11","bun-types":"latest","typescript":"^5.3.3"},"module":"dist/server.js","exports":{".":{"import":"./dist/server.js","types":"./dist/server.d.ts"},"./site":{"import":"./dist/lib/scraper.js","types":"./dist/lib/scraper.d.ts"},"./token":{"import":"./dist/lib/coingecko.js","types":"./dist/lib/coingecko.d.ts"},"./embeddings":{"import":"./dist/lib/embeddings.js","types":"./dist/lib/embeddings.d.ts"},"./tweets":{"import":"./dist/lib/tweets.js","types":"./dist/lib/tweets.d.ts"}},"keywords":["blockchain","scraper","bera","cli","api"],"_id":"@agixbt/elizascraper@1.0.1","gitHead":"0e1b52e5919b454adeb30acca933b2fc7920fdb0","_nodeVersion":"18.19.1","_npmVersion":"10.2.4","dist":{"integrity":"sha512-QMuTKzg5v28N4lorTxxJociHE5thOHKyLAlPRhTcsB3Q5SLYYS87wEQ4I4dvbfHJegeCLzkjaRt8VNWIO+oblA==","shasum":"e80dff9258982d6ab7bc20aa85c3f4175bdc4a8b","tarball":"https://registry.npmjs.org/@agixbt/elizascraper/-/elizascraper-1.0.1.tgz","fileCount":20,"unpackedSize":2938304,"signatures":[{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEUCIQDBojpUSwOlkzDCZRA24r5GZK8V2C3IfRhtqB5DwTXv7QIgVKKh/drGaXXTq46Cc7w6/tmzOUpOE7nmeIihYDpdGL4="}]},"_npmUser":{"name":"agixbt","email":"foragixbt@proton.me"},"directories":{},"maintainers":[{"name":"agixbt","email":"foragixbt@proton.me"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/elizascraper_1.0.1_1741629343306_0.3916570966571642"},"_hasShrinkwrap":false}},"time":{"created":"2025-03-10T17:53:03.295Z","modified":"2025-03-10T17:55:43.747Z","1.0.0":"2025-03-10T17:53:03.636Z","1.0.1":"2025-03-10T17:55:43.578Z"},"keywords":["blockchain","scraper","bera","cli","api"],"description":"Awesome Scraper for eliza, scrape docs, tweets, and tokens","maintainers":[{"name":"agixbt","email":"foragixbt@proton.me"}],"readme":"# Eliza Scraper\n\nA powerful scraping and RAG (Retrieval Augmented Generation) toolkit for collecting, processing, and serving blockchain ecosystem data. Built with Bun, TypeScript, and modern vector search capabilities.\n\n## Features\n\n- 🤖 Automated Twitter data scraping with engagement filtering\n- 📊 Real-time token price and market data tracking via CoinGecko\n- 📑 Documentation site scraping with caching\n- 🔄 Configurable cron jobs for automated data collection\n- 🔍 Vector search with time-decay relevance scoring\n- 🚀 REST API with Swagger documentation\n\n## Architecture\n\n### RAG Pipeline for Tweet Processing\n\n```mermaid\ngraph TD\n    A[Twitter] --> B[Apify Scraper]\n    B --> C[Filters]\n    C --> D[Text Processing]\n    D --> E[OpenAi Embedding]\n    E --> F[Qdrant Vector DB]\n    F --> G[Search API]\n    \n    subgraph \"filters\"\n    C --> H[Min Replies]\n    C --> I[Min Retweets]\n    C --> J[Search Tags]\n    C --> K[Kaito Most Mindshare handles]\n    end\n    \n    subgraph \"Vector Search\"\n    F --> L[Semantic Search]\n    end \n```\n\n### System Architecture\n\n```mermaid\ngraph TB\n    CLI[CLI Interface] --> Services\n    API[REST API] --> Services\n    \n    subgraph \"Services\"\n    Tweets[Tweet Service]\n    Tokens[Token Service]\n    Docs[Site Service]\n    end\n    \n    subgraph \"Storage\"\n    Qdrant[(Qdrant Vector DB)]\n    Postgres[(PostgreSQL)]\n    end\n    \n    Services --> Storage\n```\n\n## Scraper Quick Start\n\n1. Install globally:\n```bash\nnpm i @agixbt/elizascraper\n```\n\n2. Create `.env` file:\n```env\n# Required APIs\nAPIFY_API_KEY=your_apify_key\nCOINGECKO_API_KEY=your_coingecko_key\nOPENAI_API_KEY=\n\n# Database\nDATABASE_URL=postgresql://user:password@localhost:5432/token_scraper\n\n# Vector DB\nQDRANT_URL=http://localhost:6333\nQDRANT_COLLECTION_NAME=twitter_embeddings\n\n# Optional\nTWITTER_SEARCH_TAGS=berachain,bera\nAPP_TOPICS=berachain launch,berachain token\n```\n\n3. Start required services:\n```bash\ndocker-compose up -d\n```\n\n## CLI Commands\n\n### Start API Server\n```bash\nelizascraper server --port <number>\n\n```\n\n### Tweet Scraping Service\n```bash\nelizascraper tweets-cron [options]\n\nOptions:\n--port Port number (default: 3000)\n--tags Custom search tags, can also pass twitter search strings, check out https://github.com/igorbrigadir/twitter-advanced-search (comma-separated)\n--start Start date (YYYY-MM-DD)\n--end End date (YYYY-MM-DD)\n--min-replies Minimum replies threshold (default: 4)\n--retweets Minimum retweets threshold (default: 2)\n--max-items Maximum tweets to fetch (default: 200)\n```\nDisclaimer: make sure you don't run the cron job in a small time interval, with start and end date less than 3 days, fetching less than 50 tweets, would lead to a ban shortly on your apify key.\n\nTo prevent ban, same search query, ran multiple times inside an hour would be skipped over.\n\n### Tracking Latest Token Data\n```bash\nelizascraper token-cron [options]\n\nOptions:\n-p, --port <number> Port to listen on (default: 3008)\n-i, --interval <number> Interval in minutes (default: 60)\n--currency <string> Currency to track (default: 'usd')\n--cg_category <string> CoinGecko category (default: 'berachain-ecosystem')\n```\n\n\n### Documentation Scraping Service\n```bash\nelizascraper site-cron [options]\n\nOptions:\n-p, --port <number> Port to listen on (default: 3010)\n-i, --interval <number> Interval in minutes (default: 60)\n-l, --link <string> URL to scrape (default: 'https://docs.berachain.com')\n```\n\n## API Documentation\n\n### Tweets API\n\n#### GET /tweets/search\nSearch for tweets based on semantic similarity.\n\n**Query Parameters:**\n- `searchString` (required): Text to search for semantically similar tweets\n\n**Response Schema:**\n```typescript\n{\n  status: \"success\" | \"error\",\n  results: {\n    tweet: {\n      id: string,\n      text: string,\n      fullText: string,\n      author: {\n        name: string,\n        userName: string,\n        profilePicture: string\n      },\n      createdAt: string,\n      retweetCount: number,\n      replyCount: number,\n      likeCount: number\n    },\n    combinedScore: number,\n    originalScore: number,\n    date: string\n  }[]\n}\n```\n\n\n#### GET /tweets/random\nGet random tweets by predefined topics.\n\n**Response Schema:**\n```typescript\n{\n  topic: string,\n  tweet: {\n    tweet: {\n      id: string,\n      text: string,\n      fullText: string,\n      author: {\n        name: string,\n        userName: string,\n        profilePicture: string\n      },\n      createdAt: string,\n      retweetCount: number,\n      replyCount: number,\n      likeCount: number\n    },\n    combinedScore: number,\n    originalScore: number,\n    date: string\n  }[]\n}\n```\n\n### Tokens API\n\n#### GET /tokens\nGet token market data.\n\n**Query Parameters:**\n- `symbol` (optional): Token symbol to filter results\n\n**Response Schema:**\n```typescript\n{\n  status: \"success\" | \"error\",\n  results: {\n    id: string,\n    symbol: string,\n    name: string,\n    image: string,\n    currentPrice: number,\n    marketCap: number,\n    marketCapRank: number,\n    fullyDilutedValuation: number,\n    totalVolume: number,\n    high24h: number,\n    low24h: number,\n    priceChange24h: number,\n    priceChangePercentage24h: number,\n    marketCapChange24h: number,\n    marketCapChangePercentage24h: number,\n    circulatingSupply: number,\n    totalSupply: number,\n    maxSupply: number,\n    ath: number,\n    athChangePercentage: number,\n    athDate: string,\n    atl: number,\n    atlChangePercentage: number,\n    atlDate: string,\n    lastUpdated: string\n  }\n}\n```\n\n\n### Documentation API\n\n#### GET /docs/scrape\nScrape and cache documentation from a given URL.\n\n**Query Parameters:**\n- `url` (required): URL of the documentation site to scrape\n\n**Response Schema:**\n```typescript\n{\n  status: \"success\" | \"error\",\n  data: {\n    title: string,\n    last_updated: string,\n    total_sections: number,\n    sections: {\n      topic: string,\n      source_url: string,\n      overview: string,\n      subsections: {\n        title: string,\n        content: string\n      }[]\n    }[]\n  },\n  source: \"cache\" | \"fresh\"\n} | {\n  status: \"error\",\n  message: string\n}\n```\n\n**Cache Behavior:**\n- Documentation is cached for 7 days\n- Returns cached version if available and not expired\n- Automatically refreshes cache if data is older than 7 days\n\n\n### Error Handling\n\nAll routes use a standardized error response format:\n\n```typescript\n{\n  status: \"error\",\n  code: string,\n  message: string\n}\n```\n\n\n## Contributing\n\nContributions are welcome! Please read our contributing guidelines before submitting pull requests.\n\n## License\n\nMIT","readmeFilename":"README.md"}