{"_id":"crawlbase","_rev":"1-1bf76f8a028d043dfa4a948910422a12","name":"crawlbase","description":"Dependency free module for scraping and crawling websites using [Crawlbase](https://crawlbase.com) API","dist-tags":{"latest":"1.0.2"},"versions":{"1.0.0":{"name":"crawlbase","version":"1.0.0","keywords":["scraping","crawling","scraper","scrape","crawler","crawlbase","scraping-websites","scraping-framework","crawlbase-api","leads","leads-api"],"author":{"name":"Crawlbase","email":"info@crawlbase.com"},"license":"Apache-2.0","_id":"crawlbase@1.0.0","maintainers":[{"name":"crawlbase","email":"info@crawlbase.com"}],"homepage":"https://github.com/crawlbase-source/crawlbase-node","bugs":{"url":"https://github.com/crawlbase-source/crawlbase-node/issues"},"dist":{"shasum":"78491ad345805b6e608c910505ae445716bbd416","tarball":"https://registry.npmjs.org/crawlbase/-/crawlbase-1.0.0.tgz","fileCount":25,"integrity":"sha512-fi2ICmJd58FRN836wO6k8tWTeBFtTQ5RPOeAw5QnsltzRRsi6NVc+KR/guZICb8AnBVuwnQDjmVTWzEOeJavgQ==","signatures":[{"sig":"MEQCIDARUbz76fHfI/lHvkMrO48/dW0MqEWGbqs1rkFBuOGpAiBibPtGIHWZnfPXP6a3WbckSjr962xIR08ImUndOo+DKQ==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":38881},"main":"index.js","types":"./index.d.ts","gitHead":"97692ec83a68f1e410ccb4ddf3d9419953764a62","scripts":{"tsc":"rm -f index.d.ts && rm -f src/*.d.ts && tsc --allowJs -d --emitDeclarationOnly index.js","test":"node test.js","prepare":"husky install"},"_npmUser":{"name":"crawlbase","email":"info@crawlbase.com"},"repository":{"url":"git+ssh://git@github.com/crawlbase-source/crawlbase-node.git","type":"git"},"_npmVersion":"8.11.0","description":"Dependency free module for scraping and crawling websites using [Crawlbase](https://crawlbase.com) API","directories":{},"_nodeVersion":"16.15.1","_hasShrinkwrap":false,"devDependencies":{"husky":"^7.0.4","eslint":"^8.11.0","prettier":"^2.5.1","typescript":"^4.6.2","@commitlint/cli":"^16.2.1","eslint-config-prettier":"^8.5.0","@commitlint/config-conventional":"^16.2.1"},"_npmOperationalInternal":{"tmp":"tmp/crawlbase_1.0.0_1688375953609_0.09073807196704098","host":"s3://npm-registry-packages"}},"1.0.2":{"name":"crawlbase","version":"1.0.2","description":"Dependency free module for scraping and crawling websites using [Crawlbase](https://crawlbase.com) API","main":"index.js","scripts":{"test":"node test.js","tsc":"rm -f index.d.ts && rm -f src/*.d.ts && tsc --allowJs -d --emitDeclarationOnly index.js","prepare":"husky install"},"repository":{"type":"git","url":"git+ssh://git@github.com/crawlbase-source/crawlbase-node.git"},"author":{"name":"Crawlbase","email":"info@crawlbase.com"},"keywords":["scraping","crawling","scraper","scrape","crawler","crawlbase","scraping-websites","scraping-framework","crawlbase-api","leads","leads-api"],"bugs":{"url":"https://github.com/crawlbase-source/crawlbase-node/issues"},"license":"Apache-2.0","homepage":"https://github.com/crawlbase-source/crawlbase-node","devDependencies":{"@commitlint/cli":"^19.3.0","@commitlint/config-conventional":"^19.2.2","eslint":"^9.5.0","eslint-config-prettier":"^9.1.0","eslint-plugin-prettier":"^5.1.3","husky":"^9.0.11","prettier":"^3.3.2","typescript":"^5.4.5"},"types":"./index.d.ts","_id":"crawlbase@1.0.2","gitHead":"31b0d967521cb2018f49e82cd0f30242bf9db95f","_nodeVersion":"18.20.2","_npmVersion":"10.5.0","dist":{"integrity":"sha512-1R3BAciKP2zEqny2kgsSE02qCmlUmUKByNf4PHa8O4YLqWq6TZMgrkRW5iWX4tOevXdPfiuup0LxsfzomtvJfw==","shasum":"889176caf9b5b52f472ff6824e13d9080f91658d","tarball":"https://registry.npmjs.org/crawlbase/-/crawlbase-1.0.2.tgz","fileCount":25,"unpackedSize":39190,"signatures":[{"keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA","sig":"MEUCIC4Iv4zV3AFqb9ML3xVXKAJdBogzg4SWUo8XR4BihFyGAiEAnIr0dazj/TUrOeuRVQbP1OtHV9ex+gGoWP8JNKtvTok="}]},"_npmUser":{"name":"crawlbase","email":"info@crawlbase.com"},"directories":{},"maintainers":[{"name":"crawlbase","email":"info@crawlbase.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages","tmp":"tmp/crawlbase_1.0.2_1718850939979_0.28927182054772205"},"_hasShrinkwrap":false}},"time":{"created":"2023-07-03T09:19:13.609Z","modified":"2024-06-20T02:35:40.362Z","1.0.0":"2023-07-03T09:19:13.779Z","1.0.2":"2024-06-20T02:35:40.187Z"},"maintainers":[{"name":"crawlbase","email":"info@crawlbase.com"}],"author":{"name":"Crawlbase","email":"info@crawlbase.com"},"repository":{"type":"git","url":"git+ssh://git@github.com/crawlbase-source/crawlbase-node.git"},"keywords":["scraping","crawling","scraper","scrape","crawler","crawlbase","scraping-websites","scraping-framework","crawlbase-api","leads","leads-api"],"license":"Apache-2.0","homepage":"https://github.com/crawlbase-source/crawlbase-node","bugs":{"url":"https://github.com/crawlbase-source/crawlbase-node/issues"},"readme":"# Crawlbase node\n\nDependency free module for scraping and crawling websites using [Crawlbase](https://crawlbase.com) API\n\n## Installation\n\nInstall using npm\n\n```javascript\nnpm i crawlbase\n```\n\nRequire the necessary API class in your project.  \nYou can get your [Crawlbase free token from here](https://crawlbase.com/signup).\n\n```javascript\nconst { CrawlingAPI, ScraperAPI, LeadsAPI, ScreenshotsAPI } = require('crawlbase');\n```\n\n## Crawling API usage\n\nInitialize with one of your account tokens, either normal or javascript token. Then make get or post requests accordingly.\n\n```javascript\nconst api = new CrawlingAPI({ token: 'YOUR_TOKEN' });\n```\n\n### GET requests\n\nPass the url that you want to scrape plus any options from the ones available in the [API documentation](https://crawlbase.com/dashboard/docs).\n\n```javascript\napi.get(url, options);\n```\n\nExample:\n\n```javascript\napi.get('https://www.facebook.com/britneyspears').then(response => {\n  if (response.statusCode === 200) {\n    console.log(response.body);\n  }\n}).catch(error => console.error);\n```\n\nYou can pass any options from Crawlbase API.\n\nExample:\n\n```javascript\napi.get('https://www.reddit.com/r/pics/comments/5bx4bx/thanks_obama/', {\n  userAgent: 'Mozilla/5.0 (Windows NT 6.2; rv:20.0) Gecko/20121202 Firefox/30.0',\n  format: 'json'\n}).then(response => {\n  if (response.statusCode === 200) {\n    console.log(response.body);\n  }\n}).catch(error => console.error);\n```\n\n### POST requests\n\nPass the url that you want to scrape, the data that you want to send which can be either a json or a string, plus any options from the ones available in the [API documentation](https://crawlbase.com/dashboard/docs).\n\n```javascript\napi.post(url, data, options);\n```\n\nExample:\n\n```javascript\napi.post('https://producthunt.com/search', { text: 'example search' }).then(response => {\n  if (response.statusCode === 200) {\n    console.log(response.body);\n  }\n}).catch(error => console.error);\n```\n\nYou can send the data as application/json instead of x-www-form-urlencoded by setting options `postType` as json.\n\n```javascript\napi.post('https://httpbin.org/post', { some_json: 'with some value' }, { postType: 'json' }).then(response => {\n  if (response.statusCode === 200) {\n    console.log(response.body);\n  }\n}).catch(error => console.error);\n```\n\n### PUT requests\n\nPass the url that you want to scrape, the data that you want to send which can be either a json or a string, plus any options from the ones available in the [API documentation](https://crawlbase.com/dashboard/docs).\n\n```javascript\napi.put(url, data, options);\n```\n\nExample:\n\n```javascript\napi.put('https://producthunt.com/search', { text: 'example search' }).then(response => {\n  if (response.statusCode === 200) {\n    console.log(response.body);\n  }\n}).catch(error => console.error);\n```\n\n### Javascript requests\n\nIf you need to scrape any website built with Javascript like React, Angular, Vue, etc. You just need to pass your javascript token and use the same calls. Note that only `.get` is available for javascript and not `.post`.\n\n```javascript\nconst api = new CrawlingAPI({ token: 'YOUR_JAVASCRIPT_TOKEN' });\n```\n\n```javascript\napi.get('https://www.nfl.com').then(response => {\n  if (response.statusCode === 200) {\n    console.log(response.body);\n  }\n}).catch(error => console.error);\n```\n\nSame way you can pass javascript additional options.\n\n```javascript\napi.get('https://www.freelancer.com', { pageWait: 5000 }).then(response => {\n  if (response.statusCode === 200) {\n    console.log(response.body);\n  }\n}).catch(error => console.error);\n```\n\n### Original status and PC status\n\nYou can always get the original status and crawlbase status from the response. Read the [Crawlbase documentation](https://crawlbase.com/dashboard/docs) to learn more about those status.\n\n```javascript\napi.get('https://craiglist.com').then(response => {\n  console.log(response.originalStatus, response.cbStatus);\n}).catch(error => console.error);\n```\n\n## Scraper API usage\n\nInitialize the Scraper API and use it in the same way as the Crawling API (see above). Use it with your normal token.\n\n```javascript\nconst api = new ScraperAPI({ token: 'YOUR_TOKEN' });\n\napi.get('https://www.amazon.com/Halo-SleepSack-Swaddle-Triangle-Neutral/dp/B01LAG1TOS').then(response => {\n  if (response.statusCode === 200) {\n    console.log(response.json);\n  }\n}).catch(error => console.error);\n```\n\n## Leads API usage\n\nInitialize with your Leads API token and call the `getFromDomain` method.\n\n```javascript\nconst api = new LeadsAPI({ token: 'YOUR_TOKEN' });\n\napi.getFromDomain('somesite.com').then(response => {\n  console.log(response.leads);\n});\n```\n\n## Screenshots API usage\n\nInitialize with your Screenshots API token and call the `get` method, then do whatever you need with the binary content. For example save it in a file.\n\nYou can pass any of the [available parameters](https://crawlbase.com/docs/screenshots-api/parameters/)\n\n```javascript\nconst api = new ScreenshotsAPI({ token: 'YOUR_TOKEN' });\n\napi.get('https://www.amazon.com').then(response => {\n  fs.writeFileSync('amazon.jpg', response.body, { encoding: 'binary' });\n});\n\n// Example with parameters\napi.get('https://www.amazon.com', { device: 'mobile' }).then(response => {\n  fs.writeFileSync('amazon-mobile.jpg', response.body, { encoding: 'binary' });\n});\n```\n\nIf you have questions or need help using the library, please open an issue or [contact us](https://crawlbase.com/contact).\n\n---\n\nCopyright 2024 Crawlbase\n","readmeFilename":"README.md"}