{"_id":"spidey","_rev":"26-fcb97fb3e3091c9513d76b9181338ad5","time":{"modified":"2023-05-06T13:15:20.040Z","created":"2023-05-02T20:08:32.172Z","0.1.0":"2013-11-01T12:05:30.213Z","0.1.1":"2013-11-02T13:00:58.144Z","0.1.2":"2013-11-03T15:01:23.265Z","0.1.3":"2013-11-19T20:52:29.669Z","0.2.0":"2013-11-23T11:29:16.986Z","0.2.1":"2013-11-25T19:47:34.069Z","0.2.2":"2013-12-29T23:10:17.618Z","1.0.0":"2023-05-02T20:08:32.532Z","1.0.1":"2023-05-02T22:23:14.474Z","1.0.2":"2023-05-03T09:19:27.186Z","1.0.3":"2023-05-03T09:21:14.289Z","1.0.4":"2023-05-03T10:46:19.940Z","1.0.5":"2023-05-04T09:59:36.895Z","1.0.6":"2023-05-05T11:40:00.927Z","1.0.7":"2023-05-06T13:15:19.944Z"},"name":"spidey","dist-tags":{"latest":"1.0.7"},"versions":{"1.0.0":{"name":"spidey","version":"1.0.0","description":"Robust web spider for Node.js","main":"dist/index.js","types":"dist/index.d.ts","scripts":{"test":"echo \"Error: no test specified\" && exit 1","build":"tsc","format":"prettier --write \"src/**/*.ts\"","lint":"tslint -p tsconfig.json","prepare":"npm run build","prepublishOnly":"npm run lint","preversion":"npm run lint","version":"npm run format && git add -A src","postversion":"git push && git push --tags"},"repository":{"type":"git","url":"git+https://github.com/asad-haider/spidey.git"},"keywords":["crawler","spider","data miner","miner","scraper","web scraper","web crawler","web spider","web data miner","web data scraper","web data crawler","web data spider"],"author":{"name":"Asad Haider"},"license":"ISC","bugs":{"url":"https://github.com/asad-haider/spidey/issues"},"homepage":"https://github.com/asad-haider/spidey#readme","devDependencies":{"@types/node":"^18.16.2","@types/xmldom":"^0.1.31","prettier":"^2.8.8","ts-node":"^10.9.1","tslint":"^6.1.3","tslint-config-prettier":"^1.18.0","typescript":"^5.0.4"},"dependencies":{"axios":"^1.3.6","cheerio":"^1.0.0-rc.12","winston":"^3.8.2","xmldom":"^0.6.0","xpath":"^0.0.32"},"gitHead":"8c24c812980fe473c1a8e10a7ce945b885264ec9","_id":"spidey@1.0.0","_nodeVersion":"16.15.1","_npmVersion":"8.11.0","dist":{"integrity":"sha512-sg3XRKI79n+O0tDJ2qlPVXoWcvQv6nopNbQYkj1FUQNuG63kbFazbx7CviPsR2zGyKJgQEgCNKoGEXB/xrTwrA==","shasum":"0318d54989cd33a8406f5a92c82dbacc4bb2ac19","tarball":"https://registry.npmjs.org/spidey/-/spidey-1.0.0.tgz","fileCount":16,"unpackedSize":33800,"signatures":[{"keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA","sig":"MEYCIQDud5tQ5RyeyHmUIGw0IAtufKO6fn1hDgyn+k9lC9MwJAIhANUNyq7gdH07lgKH+q2jgwzrQF54JW/NtKv0XcW9SVs+"}],"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v4.10.10\r\nComment: https://openpgpjs.org\r\n\r\nwsFzBAEBCAAGBQJkUW3AACEJED1NWxICdlZqFiEECWMYAoorWMhJKdjhPU1b\r\nEgJ2VmqHUw/9HKhJpee2PZ6opy3P473mdO5pys7j4y7E/dBRjvbn9Nondqjb\r\ncMwbjsIGJL8A3PkXW0W5izsnZxWE/xtYFsoAO2gqWzOPJXqtnEPK7J/wB2tT\r\nm7rFpMVUoOm2eJqxv+iwYuPF40C0wMPsh0/dmDyDjanJmfuHS1RldhH9gi3M\r\nB7tzkYc7IhP7k+IPL02lHbfiDSrcYHkJmkqOIhREiXwTMlmlnTLbHjwQoGzw\r\nRuAtd+u+d186wtHaETZ4DjV94bhcl2pW2r9affraC3HCtGQeOf+wtW0bsQ9K\r\njjDoisjI+Fg38QBc+oxTeCPkNVRiQPUi8E8UBNEWpQ9F2LVFYPZyU0rr2qor\r\ni9B5f83AOz6ACD9w8x+CUgXQ6TUBxOei+ZIX9jpEmD1M8zJZfHXwgHOV2P+2\r\nJses99jeYj5LgZY/sULzZbwBBMxcpTop4jlUb1N/qYKz2/aUa7689z4vZ+k3\r\nHIMSv+8mSAbXK1Hpn1Fem9N8067qqRKQYmrCzdhtJxBEzHyohvlS5c1BUIlp\r\nkDEFGsUCqKqSKv8uS7p6qtd4djwMXtPICTpV2JJZUtsPwBW/lMs8P11hvYAT\r\nXNhIDniub5QhpxbwGaF0byQJYxmVN3GIfYHC0q+q8zQb/YLV/b3jrWkJb7+j\r\nq/epBiGWoOi9K4M6jxnvHdjTLGo179wS6sU=\r\n=gbLO\r\n-----END PGP SIGNATURE-----\r\n"},"_npmUser":{"name":"smasadhaider","email":"smasadhaider@gmail.com"},"directories":{},"maintainers":[{"name":"smasadhaider","email":"smasadhaider@gmail.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages","tmp":"tmp/spidey_1.0.0_1683058112173_0.9083045239953187"},"_hasShrinkwrap":false},"1.0.1":{"name":"spidey","version":"1.0.1","description":"Robust web spider for Node.js","main":"dist/index.js","types":"dist/index.d.ts","scripts":{"test":"echo \"Error: no test specified\" && exit 1","build":"tsc","format":"prettier --write \"src/**/*.ts\"","lint":"tslint -p tsconfig.json","prepare":"npm run build","prepublishOnly":"npm run lint","preversion":"npm run lint","version":"npm run format && git add -A src","postversion":"git push && git push --tags"},"repository":{"type":"git","url":"git+https://github.com/asad-haider/spidey.git"},"keywords":["crawler","spider","data miner","miner","scraper","web scraper","web crawler","web spider","web data miner","web data scraper","web data crawler","web data spider"],"author":{"name":"Asad Haider"},"license":"ISC","bugs":{"url":"https://github.com/asad-haider/spidey/issues"},"homepage":"https://github.com/asad-haider/spidey#readme","devDependencies":{"@types/node":"^18.16.2","@types/xmldom":"^0.1.31","prettier":"^2.8.8","ts-node":"^10.9.1","tslint":"^6.1.3","tslint-config-prettier":"^1.18.0","typescript":"^5.0.4"},"dependencies":{"axios":"^1.3.6","cheerio":"^1.0.0-rc.12","winston":"^3.8.2","xmldom":"^0.6.0","xpath":"^0.0.32"},"gitHead":"b70e560e06556df38edb6dc99758e9222e528d09","_id":"spidey@1.0.1","_nodeVersion":"16.15.1","_npmVersion":"8.11.0","dist":{"integrity":"sha512-bWj/KPjEXeAKUA8O5w/ybcJojwsS6q5NqMODrxQLS0HMrhFeAf6Bgzivo9XErMHJ9Nv0olRnmvlt5+nONXKExA==","shasum":"528406117a62c0798c826e4f3fa4959e90efa374","tarball":"https://registry.npmjs.org/spidey/-/spidey-1.0.1.tgz","fileCount":16,"unpackedSize":37314,"signatures":[{"keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA","sig":"MEQCIGtKTws3jt3TuEpQvjTDCS+Yoo0AYTxUg0BneUTQkcz/AiBCJ+oa9KUoYueRQxYG/YHHWIUabjQ55SGXLimahHA6sA=="}],"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v4.10.10\r\nComment: https://openpgpjs.org\r\n\r\nwsFzBAEBCAAGBQJkUY1SACEJED1NWxICdlZqFiEECWMYAoorWMhJKdjhPU1b\r\nEgJ2VmpzcBAAlRCNjQi+/YgwrQLiBvuhJXkWq+x8/NxJvbTzPuOmAV5gLTVK\r\nsD+hljZngM5vkoJPrgHGYC4jPNScSOtulZJgBzNPRzGTekkjcW4rrXmrtuOy\r\nY0yJMhf/5DoVWp+d6Iy7se3tcjfrNgH+7QXeFOm1q7KFvCk5dAdqELbXdVj7\r\n4MIXOendPRRgNZ0YTg1K3y265N0T93U1uaaWdelENVApapdJZyYq43siB3Wm\r\nW9vhyjGGyLQjmHnMx9PqRyG9hJEkkyrrJKKOuf7fWET3XezO9n/WngP+I+t0\r\nxuis44JCis6SgSS82T5FxXCmpRLh7FfQblF2miEP4TiVOUw26zFAcksA9tIA\r\n5VlZm5B6WrhrnKsCNFLdjlTQjzVte8WU5PEJMRKeoEIYx0d/48V5V7u8RRJo\r\nu/XbuvBIop0/NxUqeRyaDM+NYtQzXZ+nY2YP07hsfHzQPzghxn/WV7ykKerR\r\nLk21M1UsqIsh2CBrcsVxPEk45lzyMlycsoDblAEEJr9VVzJugqYLKPi+XxvW\r\nYzzCIKN4+/2QZe6GYK8Dr4E7v9CEpmQ8vIU9lVvORGxCzJxFawr5Twec0Up0\r\nzt9KL3YgbtaoYOiRJJ4PKViZtF979znnFYju6nobRSYprE3TMIrAZBKOH6Up\r\neVhJnud+rqssFIg/c5keXA2cQjrEjQXI1m8=\r\n=GhIv\r\n-----END PGP SIGNATURE-----\r\n"},"_npmUser":{"name":"smasadhaider","email":"smasadhaider@gmail.com"},"directories":{},"maintainers":[{"name":"smasadhaider","email":"smasadhaider@gmail.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages","tmp":"tmp/spidey_1.0.1_1683066194221_0.3007180438719137"},"_hasShrinkwrap":false},"1.0.2":{"name":"spidey","version":"1.0.2","description":"Robust web spider for Node.js","main":"dist/index.js","types":"dist/index.d.ts","scripts":{"test":"echo \"Error: no test specified\" && exit 1","build":"tsc","format":"prettier --write \"src/**/*.ts\"","lint":"tslint -p tsconfig.json","prepare":"npm run build","prepublishOnly":"npm run lint","preversion":"npm run lint","version":"npm run format && git add -A src","postversion":"git push && git push --tags"},"repository":{"type":"git","url":"git+https://github.com/asad-haider/spidey.git"},"keywords":["crawler","spider","data miner","miner","scraper","web scraper","web crawler","web spider","web data miner","web data scraper","web data crawler","web data spider"],"author":{"name":"Asad Haider"},"license":"ISC","bugs":{"url":"https://github.com/asad-haider/spidey/issues"},"homepage":"https://github.com/asad-haider/spidey#readme","devDependencies":{"@types/node":"^18.16.2","@types/xmldom":"^0.1.31","prettier":"^2.8.8","ts-node":"^10.9.1","tslint":"^6.1.3","tslint-config-prettier":"^1.18.0","typescript":"^5.0.4"},"dependencies":{"axios":"^1.3.6","cheerio":"^1.0.0-rc.12","winston":"^3.8.2","xmldom":"^0.6.0","xpath":"^0.0.32"},"gitHead":"fde9fbcb2814362dfa48889ff6b74a9e27e49eb8","_id":"spidey@1.0.2","_nodeVersion":"16.15.1","_npmVersion":"8.11.0","dist":{"integrity":"sha512-q+Sik6SCGUtJX5vQGXhkkt8zyu6Esjt9BWuMhIqHCmpV9jV9h7Q+cOS7rCuDCMeJRUwhfPXZFJIrFcdAXsWpAQ==","shasum":"8ad2a4f5e6f48e15e943ec38c32d14c9c551352e","tarball":"https://registry.npmjs.org/spidey/-/spidey-1.0.2.tgz","fileCount":16,"unpackedSize":40875,"signatures":[{"keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA","sig":"MEUCIH0FvizUdjhm5OlqUaF94ITO8iX4v/5lUTIIWgePhCa2AiEAw4fIqEJeCV98ekSvLd4dv3zv3s9aeLmIx/o4hwBJRPg="}],"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v4.10.10\r\nComment: https://openpgpjs.org\r\n\r\nwsFzBAEBCAAGBQJkUicfACEJED1NWxICdlZqFiEECWMYAoorWMhJKdjhPU1b\r\nEgJ2Vmp0kw/9G0OeIkplxoK3ueY2uoox8/uuSfludbCLbTyBxG5Jz8DykWwP\r\ntYgoXlcq30ECCGccBGi1XesyrejpCJ5iqwISayP6i8/D8J8sZ6vTNPnLWiYf\r\nOwEmblJO2tueC2/OtUjxo1nbr+ZGA+m9h2bFMWC/rvAGoAro8IYtWNZ6oGgP\r\nupLkkXXrWcXYpIOGF256ip8Zmu1mj1Bjk9HWwCMnt0OETPRnJ9gqiE9J0pGw\r\nOdVAD5Uphkjd6fRrF1FLvwU8UcL++dbN4SU/QL7/YhcFHTLiOmuBA3lEPc1A\r\nupLGwRQC3Viq5yOJyOBg3hhZtTjdliMu3EC7OoeaRnNmjeYBpmPoJWxlRzd6\r\n+nLk72lG21R4sTSR4C8yoMBUUzLBPZpUWIYv19SYI3nsx/+sCgS3W+uN9B3N\r\nTEqP6FH/3zaPJTxmhFy0jE1gYiH2LLQjE3Zgw5hMTm4H7Wzojh716+aBkj5e\r\nMrH6RIgP6zqW1L6HvJ5yH+PSS1IXGBAuEmU5XTb4OEudy6EHvTR1XjhxgojS\r\na1gjUyQCAOToInOcD3V+jSKtqe4Dq7Y6gpV5MZmcUbxpGChq+2IkxnMhZEBT\r\nkAyhGTi2CZlvlWUR7qnUYyss9SMdhvjttPg0uW0lxlmIFFJ9TLgeDjbDx8IA\r\n886RLwbdwIDfTQuYXarov7wP0T1et8TeDqo=\r\n=HSGS\r\n-----END PGP SIGNATURE-----\r\n"},"_npmUser":{"name":"smasadhaider","email":"smasadhaider@gmail.com"},"directories":{},"maintainers":[{"name":"smasadhaider","email":"smasadhaider@gmail.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages","tmp":"tmp/spidey_1.0.2_1683105566990_0.27088247379232255"},"_hasShrinkwrap":false},"1.0.3":{"name":"spidey","version":"1.0.3","description":"Robust web spider for Node.js","main":"dist/index.js","types":"dist/index.d.ts","scripts":{"test":"echo \"Error: no test specified\" && exit 1","build":"tsc","format":"prettier --write \"src/**/*.ts\"","lint":"tslint -p tsconfig.json","prepare":"npm run build","prepublishOnly":"npm run lint","preversion":"npm run lint","version":"npm run format && git add -A src","postversion":"git push && git push --tags"},"repository":{"type":"git","url":"git+https://github.com/asad-haider/spidey.git"},"keywords":["crawler","spider","data miner","miner","scraper","web scraper","web crawler","web spider","web data miner","web data scraper","web data crawler","web data spider"],"author":{"name":"Asad Haider"},"license":"ISC","bugs":{"url":"https://github.com/asad-haider/spidey/issues"},"homepage":"https://github.com/asad-haider/spidey#readme","devDependencies":{"@types/node":"^18.16.2","@types/xmldom":"^0.1.31","prettier":"^2.8.8","ts-node":"^10.9.1","tslint":"^6.1.3","tslint-config-prettier":"^1.18.0","typescript":"^5.0.4"},"dependencies":{"axios":"^1.3.6","cheerio":"^1.0.0-rc.12","winston":"^3.8.2","xmldom":"^0.6.0","xpath":"^0.0.32"},"gitHead":"1643d30ce93ecdc692ec82b4d8d4c282944dcb9c","_id":"spidey@1.0.3","_nodeVersion":"16.15.1","_npmVersion":"8.11.0","dist":{"integrity":"sha512-wc0flHkDlRxByzACxfD62VZc2DMRxpjpkkcP49vI2rieWcoCBltEKUiS8jXmq3ZugvbVUatf29z+eRwRi3HtjQ==","shasum":"73709e10d0491ac01b7b51e07e4f1707b14e015a","tarball":"https://registry.npmjs.org/spidey/-/spidey-1.0.3.tgz","fileCount":16,"unpackedSize":40875,"signatures":[{"keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA","sig":"MEUCIHFhOiG7zNMqZV8KNfsnsRUcVp4Kad8dvb0lDkLJcdaLAiEAmzerQJrGRVyCbeNmKbS41nkXllXTesE5boU59ADbMIo="}],"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v4.10.10\r\nComment: https://openpgpjs.org\r\n\r\nwsFzBAEBCAAGBQJkUieKACEJED1NWxICdlZqFiEECWMYAoorWMhJKdjhPU1b\r\nEgJ2Vmr92w/+Nw5vatDr5RmHgDOfBzRudprPrFQ8M3+PF2JUFM7kea3Cio2A\r\nien5Uj1tVTnM6Ljl5k0AYtfc52H+Gux54B0+IDMaBla4c8rcuC2oGc/OeDPZ\r\n0jWpcpILMT9a7MsbQMeUpDai14YfJ4aRVWDZ8/LDqH9yMqcLSdMFhUWX2m6H\r\ngv/R9kw/QgRLMRcRMUW0qtZHYDDUajwO2mWkapNoZ+tz0Bib1Om4c5j8Veao\r\nNiqY/Bn7LwEeuNNUim3orHkgwOYll6Y1f6neV0qNwJuFPQnZiaAd1AUB4pNU\r\nOYUQHm6GmLGLpaoAI3S13Ou4eAUq64ssr+r+lGjk+3h4JiHEi2cR6zGlzlAl\r\nPKjCu6qp3ejiA1NFKFAx/ektbd9PMkVG/PhLJsE3/VXVY4U6HNZuH+FKH3RW\r\nLPMMoFS+UQoKzBv+p5hWbgGe5Q+ItlPA5tv3sHzd6FRqbFcouknpxxZReobw\r\n0Guvm1sxYOFqv62Q7637s8RN40KSuuo6pIKgtgI33pbQ+BDJuAY5MJsdM348\r\nWmfv0sSuuZBf5ZYTmAYq/WaLggR0EAGL65RKUsf3rgVCHgDAc60xdNw9zdFO\r\nBLtrE/RIwGHeCzsbBgrUzQ7bMnvxNJHxdfzGHNHltgfsTL5kMVchqCX9ej+C\r\nop2Y7Tjsm0hgHbWellv66SSBbyekxGLq0l8=\r\n=+z+v\r\n-----END PGP SIGNATURE-----\r\n"},"_npmUser":{"name":"smasadhaider","email":"smasadhaider@gmail.com"},"directories":{},"maintainers":[{"name":"smasadhaider","email":"smasadhaider@gmail.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages","tmp":"tmp/spidey_1.0.3_1683105674094_0.47470289651072295"},"_hasShrinkwrap":false},"1.0.4":{"name":"spidey","version":"1.0.4","description":"Robust web spider for Node.js","main":"dist/index.js","types":"dist/index.d.ts","scripts":{"test":"echo \"Error: no test specified\" && exit 1","build":"tsc","format":"prettier --write \"src/**/*.ts\"","lint":"tslint -p tsconfig.json","prepare":"npm run build","prepublishOnly":"npm run lint","preversion":"npm run lint","version":"npm run format && git add -A src","postversion":"git push && git push --tags"},"repository":{"type":"git","url":"git+https://github.com/asad-haider/spidey.git"},"keywords":["crawler","spider","data miner","miner","scraper","web scraper","web crawler","web spider","web data miner","web data scraper","web data crawler","web data spider"],"author":{"name":"Asad Haider"},"license":"ISC","bugs":{"url":"https://github.com/asad-haider/spidey/issues"},"homepage":"https://github.com/asad-haider/spidey#readme","devDependencies":{"@types/node":"^18.16.2","@types/xmldom":"^0.1.31","prettier":"^2.8.8","ts-node":"^10.9.1","tslint":"^6.1.3","tslint-config-prettier":"^1.18.0","typescript":"^5.0.4"},"dependencies":{"axios":"^1.3.6","cheerio":"^1.0.0-rc.12","winston":"^3.8.2","xmldom":"^0.6.0","xpath":"^0.0.32"},"gitHead":"d58846d6807caca2f6bc1b5138e90f8484e7cfce","_id":"spidey@1.0.4","_nodeVersion":"18.12.1","_npmVersion":"9.6.5","dist":{"integrity":"sha512-Z5yA3JB17BcHZ2pfHoJGfUv8XVgbTVwr6i6mGCyIhbZFGw2BhlKB9xoKbJdYt3kzg2RKcg60sM2gkgzpazZjig==","shasum":"6f271ea34c5476393ed9587572ece219144958e2","tarball":"https://registry.npmjs.org/spidey/-/spidey-1.0.4.tgz","fileCount":13,"unpackedSize":38106,"signatures":[{"keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA","sig":"MEQCIEYcjA9FMySoG/zou3C+9rkGpw2HZd3Hda2m+pB8jsRBAiAmaXBXjFVMNu6wynIIKuceJeLEWRJFveF6rNXC2PpgHA=="}],"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v4.10.10\r\nComment: https://openpgpjs.org\r\n\r\nwsFzBAEBCAAGBQJkUjt7ACEJED1NWxICdlZqFiEECWMYAoorWMhJKdjhPU1b\r\nEgJ2VmrRJw//Zk8F76dE/j1K7cJcYmrktCh5E4o67EINYdCTWQ8k7e8f4MhB\r\nTN6v0i5sR//w/x5eyX4wBbrXZIXh6e6x+hWwj9VSGE5LwDSnhyHFZi2G7S29\r\nM6VtwuO3yYNCcwXF1GBiAoCUaJjuXK46PBGB5rBLsrkKMGpyVGbAFGgKT9EY\r\nhctAT7oJBVJ3OeUFPru4PU4Obz8USqF2GOmhd9uTaajuKozWybzXDxAVBRRS\r\nQ1Nywz+sC+WYE3EfzMqguZQYnLZXF4Y8zzFDCe1+bsEPfIXc7xk8aiPOP3DU\r\n4F7VUQ+rchb/jbWN+SpxbXiZMwVYu9DRUgVXhlCZ7YCQRSTTIVCNFdBVPZZI\r\nOsj5JnX/cNXy0LC1tmuax2XASypam9+TcBw6JmSYK+HIpaWO1FpaMJ97ILwz\r\nR87m0Hy5d2m/qYdHHXNMQp2iY9IaKUkzhrkoYX5pp6jB04LeGU6eIsSnNDFJ\r\n4aWxfF5yOst+rdSb+kL+jRfvsGFoMSCeSVQr8jrl2JI9X7dtUPUk9DvynhvD\r\nSEdFIU+0edYy1hvUNVkxtEHJ9HaGhaUWpAZ45kIwLfxS/eQbMY0bwtwmtkCb\r\nEI+jPA/0oMhErEZWoPxnGWazupZI8K4sKgOF0epuQMj0RO+Uelgq+F3Q9EYR\r\nll9oMCzNGOBsIiA8ItQuWMmcsqy9X8FUgE0=\r\n=svLz\r\n-----END PGP SIGNATURE-----\r\n"},"_npmUser":{"name":"smasadhaider","email":"smasadhaider@gmail.com"},"directories":{},"maintainers":[{"name":"smasadhaider","email":"smasadhaider@gmail.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages","tmp":"tmp/spidey_1.0.4_1683110779785_0.6902881336575961"},"_hasShrinkwrap":false},"1.0.5":{"name":"spidey","version":"1.0.5","description":"Robust web spider for Node.js","main":"dist/index.js","types":"dist/index.d.ts","scripts":{"test":"echo \"Error: no test specified\" && exit 1","build":"tsc","format":"prettier --write \"src/**/*.ts\"","lint":"tslint -p tsconfig.json","prepare":"npm run build","prepublishOnly":"npm run lint","preversion":"npm run lint","version":"npm run format && git add -A src","postversion":"git push && git push --tags"},"repository":{"type":"git","url":"git+https://github.com/asad-haider/spidey.git"},"keywords":["crawler","spider","data miner","miner","scraper","web scraper","web crawler","web spider","web data miner","web data scraper","web data crawler","web data spider"],"author":{"name":"Asad Haider"},"license":"ISC","bugs":{"url":"https://github.com/asad-haider/spidey/issues"},"homepage":"https://github.com/asad-haider/spidey#readme","devDependencies":{"@types/node":"^18.16.2","@types/xmldom":"^0.1.31","prettier":"^2.8.8","ts-node":"^10.9.1","tslint":"^6.1.3","tslint-config-prettier":"^1.18.0","typescript":"^5.0.4"},"dependencies":{"axios":"^1.3.6","cheerio":"^1.0.0-rc.12","winston":"^3.8.2","xmldom":"^0.6.0","xpath":"^0.0.32"},"gitHead":"06bac8a4ab2aaef424b0929f863021e895047a2a","_id":"spidey@1.0.5","_nodeVersion":"18.12.1","_npmVersion":"9.6.5","dist":{"integrity":"sha512-OUsd5u/9TnSfpzI9yj13Plqp7hjQ7eAiQX4DW7iO5/x7GbgUOf34YA1BBk8V5BD/gM0S2XtWgxkUKm90zWWwhA==","shasum":"795d94967d7016f5499e35143c5fadef9b14a6cf","tarball":"https://registry.npmjs.org/spidey/-/spidey-1.0.5.tgz","fileCount":13,"unpackedSize":38223,"signatures":[{"keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA","sig":"MEYCIQDqTZwUniJoe5IgFP8C5bMS3bBwgnK3Ow72M5Gn/F/44QIhALiXMLCsvz3UkHFV/oJnfxle+GTh1184xh7p/9LuHIu/"}],"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v4.10.10\r\nComment: https://openpgpjs.org\r\n\r\nwsFzBAEBCAAGBQJkU4IIACEJED1NWxICdlZqFiEECWMYAoorWMhJKdjhPU1b\r\nEgJ2Vmqhng//VS2DRklBhcmzt9YojQWy+mEwhfrHteTHcZrLY7FQB3fnvNdj\r\nrlNrrw5VYVh2+MBbZzwycSjXy7KZE0gwU7KYSICLCszSW8wOQe/XpUKC5i0F\r\nr/outm3vg1VWm4itrGfZK8cKW45IkKrVzXSJKa2Eq5nAp13aTiuqKHZCe4Cm\r\nuFmjTrh4ECfxne2HykdUbQk2ybRq2SZAxAfx7szK/IAL3/u4ki8ncdZB7OZP\r\nizeuHoWBequCZj/jdWGDXXD/cruqBSA4DQfnx2N0s/vygqv87htY/tMqX83u\r\n5SfTRDB96WlpEeRz4gIEz1aJeu3FESuKTyErGKMk0KRIZqYpbYeqXcccSJY1\r\nd6oipFvbIjXqKViCrlrnnv79bT9PRILePpIpc2l9Nq0nE8CQfIz38mmD5Z3Y\r\ntPCLGdikeWRDnhcCqAZsKQwYHd/WdTj6MCvdo7SQ+Y6HD/ryc9bBDSN1pTYg\r\nyvoEnUMl0WEONK5WQ2xGH8pYSFmabxEnHoxhBs7Pi952luE4gQQuKy2+DLMk\r\nVuDGlSAmOaIsT6TJ+0w14hUCIKDPz09ybF0+lGoAeVF9UYKJFXLCViWe+y4Q\r\nutye6TXhasAaBGJhY2LOFDjM8dXyIyemZIU9PsZ7XoyurCdZF76TgVunMW7E\r\n84oPY42EEAiFd9w9LyL4O51tB4iomx/bRBw=\r\n=BSkx\r\n-----END PGP SIGNATURE-----\r\n"},"_npmUser":{"name":"smasadhaider","email":"smasadhaider@gmail.com"},"directories":{},"maintainers":[{"name":"smasadhaider","email":"smasadhaider@gmail.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages","tmp":"tmp/spidey_1.0.5_1683194376774_0.1901027383472429"},"_hasShrinkwrap":false},"1.0.6":{"name":"spidey","version":"1.0.6","description":"Robust web spider for Node.js","main":"dist/index.js","types":"dist/index.d.ts","scripts":{"test":"echo \"Error: no test specified\" && exit 1","build":"tsc","format":"prettier --write \"src/**/*.ts\"","lint":"tslint -p tsconfig.json","prepare":"npm run build","prepublishOnly":"npm run lint","preversion":"npm run lint","version":"npm run format && git add -A src","postversion":"git push && git push --tags"},"repository":{"type":"git","url":"git+https://github.com/asad-haider/spidey.git"},"keywords":["crawler","spider","data miner","miner","scraper","web scraper","web crawler","web spider","web data miner","web data scraper","web data crawler","web data spider"],"author":{"name":"Asad Haider"},"license":"ISC","bugs":{"url":"https://github.com/asad-haider/spidey/issues"},"homepage":"https://github.com/asad-haider/spidey#readme","devDependencies":{"@types/node":"^18.16.2","@types/xmldom":"^0.1.31","prettier":"^2.8.8","ts-node":"^10.9.1","tslint":"^6.1.3","tslint-config-prettier":"^1.18.0","typescript":"^5.0.4"},"dependencies":{"axios":"^1.3.6","bottleneck":"^2.19.5","cheerio":"^1.0.0-rc.12","winston":"^3.8.2","xmldom":"^0.6.0","xpath":"^0.0.32"},"gitHead":"725cbfb3aa8783e1b82eb4cd29906f298bd4d035","_id":"spidey@1.0.6","_nodeVersion":"18.12.1","_npmVersion":"9.6.5","dist":{"integrity":"sha512-t0cVPGiZx9/AkGuFGcVRAxvb6/GpwgUjsXomTucvxBETItCY+4QOudvg8BoXuq2fgqjsQX9PnCW+xvesjrGB4g==","shasum":"24c0496a885859b1bbd786f751e8bf55741fa74f","tarball":"https://registry.npmjs.org/spidey/-/spidey-1.0.6.tgz","fileCount":13,"unpackedSize":40528,"signatures":[{"keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA","sig":"MEQCIF/gnOQeeJB1+0xynQaFfgv9aa8ecgNCryDgVxY8xAOFAiBAOrG7gkr2QlQz3UN0WabM6bfLhwB70MEg5PAaxOJeRQ=="}]},"_npmUser":{"name":"smasadhaider","email":"smasadhaider@gmail.com"},"directories":{},"maintainers":[{"name":"smasadhaider","email":"smasadhaider@gmail.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages","tmp":"tmp/spidey_1.0.6_1683286800783_0.265110607888984"},"_hasShrinkwrap":false},"1.0.7":{"name":"spidey","version":"1.0.7","description":"Robust web spider for Node.js","main":"dist/index.js","types":"dist/index.d.ts","scripts":{"test":"echo \"Error: no test specified\" && exit 1","build":"tsc","format":"prettier --write \"src/**/*.ts\"","lint":"tslint -p tsconfig.json","prepare":"npm run build","prepublishOnly":"npm run lint","preversion":"npm run lint","version":"npm run format && git add -A src","postversion":"git push && git push --tags"},"repository":{"type":"git","url":"git+https://github.com/asad-haider/spidey.git"},"keywords":["crawler","spider","data miner","miner","scraper","web scraper","web crawler","web spider","web data miner","web data scraper","web data crawler","web data spider"],"author":{"name":"Asad Haider"},"license":"ISC","bugs":{"url":"https://github.com/asad-haider/spidey/issues"},"homepage":"https://github.com/asad-haider/spidey#readme","devDependencies":{"@types/node":"^18.16.2","@types/xmldom":"^0.1.31","prettier":"^2.8.8","ts-node":"^10.9.1","tslint":"^6.1.3","tslint-config-prettier":"^1.18.0","typescript":"^5.0.4"},"dependencies":{"axios":"^1.3.6","bottleneck":"^2.19.5","cheerio":"^1.0.0-rc.12","winston":"^3.8.2","xmldom":"^0.6.0","xpath":"^0.0.32"},"gitHead":"ef32ba7e953afaf91d824a8e5d4fe9de50685809","_id":"spidey@1.0.7","_nodeVersion":"16.15.1","_npmVersion":"8.11.0","dist":{"integrity":"sha512-qxdNAz/2fhZTuUhMvJb/aJTvlq7Ey785vDdEE4/yyRpj4BqnSOoaa7WjWJy36OR8DkZUCUyFB6QIQASvfiqfZQ==","shasum":"75697cc7423f14917dd8bbc6dcb0061b03769424","tarball":"https://registry.npmjs.org/spidey/-/spidey-1.0.7.tgz","fileCount":19,"unpackedSize":49907,"signatures":[{"keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA","sig":"MEUCICXAxyEiuLim60OkuNGLfmqwhbY2tWvw/srB1+S2VAoTAiEAxSuZgRewlfPbsHUUNQtzU2jHE0Q/khxHNeReQ6eZEV4="}]},"_npmUser":{"name":"smasadhaider","email":"smasadhaider@gmail.com"},"directories":{},"maintainers":[{"name":"smasadhaider","email":"smasadhaider@gmail.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages","tmp":"tmp/spidey_1.0.7_1683378919736_0.9584541979638073"},"_hasShrinkwrap":false}},"maintainers":[{"name":"smasadhaider","email":"smasadhaider@gmail.com"}],"description":"Robust web spider for Node.js","homepage":"https://github.com/asad-haider/spidey#readme","keywords":["crawler","spider","data miner","miner","scraper","web scraper","web crawler","web spider","web data miner","web data scraper","web data crawler","web data spider"],"repository":{"type":"git","url":"git+https://github.com/asad-haider/spidey.git"},"author":{"name":"Asad Haider"},"bugs":{"url":"https://github.com/asad-haider/spidey/issues"},"license":"ISC","readme":"\n[![npm package](https://nodei.co/npm/spidey.png?downloads=true&downloadRank=true&stars=true)](https://nodei.co/npm/spidey/)\n\n[![NPM download][download-image]][download-url]\n[![Package Quality][quality-image]][quality-url]\n\n[quality-image]: https://packagequality.com/shield/spidey.svg\n[quality-url]: https://packagequality.com/#?package=spidey\n[download-image]: https://img.shields.io/npm/dm/spidey.svg?style=flat-square\n[download-url]: https://npmjs.org/package/spidey\n\n# Spidey - Robust Web Scraping Tool\n\nSpidey is a powerful and reliable web scraping tool that allows users to crawl any website with multiple concurrency. With Spidey, you can easily extract data from websites and store it in a structured format using custom pipelines. The tool is designed to be highly scalable and can handle large amounts of data.\n\n## Features\n\n- Easy to use API\n- Supports multiple concurrency\n- Highly scalable\n- Data parsing using XPath and CSS Selectors \n- Auto retries and error status code handling\n- Built-in data pipeline and storage format\n- Supports proxies\n- Support custom data pipelines\n- Support requests prioritization\n\n## Installation\n\nYou can install Spidey using NPM:\n\n```\nnpm install spidey\n```\n\n## Usage\n\nUsing Spidey is easy. Simply require the package and create a new instance of the `Spidey` class:\n\n```javascript\nconst { Spidey } = require('spidey');\n\nconst spidey = new Spidey({\n  concurrency: 5,\n  outputFormat: 'json',\n  outputFileName: 'output.json'\n  // other options...\n});\n\nspidey.request({\n  url: 'https://example.com',\n  // spidey options...\n}, (response) => {\n  const title = response.xpath('//*[@title=\"Google\"]')[0].data;\n  const heading = response.$('.heading').text();\n  this.save({ title, heading );  \n});\n\nspider.start();\n```\n\nSpidey can also be used by extending as Class\n\n```typescript\nclass AmazonSpidey extends Spidey {\n  constructor() {\n    super({\n      concurrency: 10,\n      retries: 5,\n      // spidey options...\n    });\n  }\n\n  headers = {\n    'user-agent':\n      'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36',\n  };\n\n  startUrls = [\n    'https://www.amazon.de/-/en/Amazon-Liquid-Soap-Refill-Fragrance/dp/B0996J4VV2',\n    'https://www.amazon.de/Dancing-Interactive-Educational-Children-Recording/dp/B0BLVKYXXQ',\n  ];\n\n  parse(response: SpideyResponse) {\n    const url = response.url;\n    const title = response.xpath('//*[@id=\"productTitle\"]/text()')[0].data.trim();\n    this.save({ url, title });\n  }\n}\n\nnew MySpidey().start();\n```\n\n## Request Options\nWhen making a request using Spidey, you can pass multiple options to customize your request.\n\n```typescript\nthis.request(\n  { \n    url, \n    headers: this.headers, \n    // request options...\n  }, \n  this.parse.bind(this)\n);\n```\n\n| Name | Type | Required | Default | Description |\n|------|------|----------|---------|-------------|\n| url | string | Yes | - | Input URL for the request |\n| method | string | No | `GET` | Request method |\n| headers | any | No | `null` | Request headers |\n| body | any | No | `null` | Request body |\n| json | boolean | No | `false` | If `true`, the response will be converted into JSON |\n| timeout | number | No | `0` | Request timeout in milliseconds. A value of `0` means no timeout |\n| meta | any | No | `{}` | A meta object that can be used to pass metadata from the request to the response function |\n| inline | boolean | No | `false` | If `true`, the response will be returned directly instead of being passed to a callback function |\n| proxy | SpdieyProxy | No | `null` | Proxy configuration |\n| proxyUrl | string | No | `null` | Proxy configuration as a URL |\n| priority | number | No | `null` | Request priority. A number between 0-9, where 0 is highest priority and 9 is lowest priority. If not specified, no priority is assigned. |\n\n\n\n## Spidey Options\nYou can pass Spidey options to a Spidey instance\n\n```javascript\nconst spidey = new Spidey({\n  // spidey options...\n});\n\n```\nOR\n\n```typescript\nclass MySpidey extends Spidey {\n  constructor() {\n    super({\n      // spidey options...\n    });\n  }\n}\n\n```\n\n| Configuration      | Type     | Description                                                                                                      | Default Value                                    |\n|--------------------|----------|------------------------------------------------------------------------------------------------------------------|--------------------------------------------------|\n| concurrency        | number   | The number of concurrent requests.                                                                              | 10                                               |\n| delay              | number   | The delay between each request.                                                                                  | 0                                                |\n| retries            | number   | The number of retries for each failed request.                                                                   | 0                                                |\n| retryStatusCode    | number[] | Spidey will retry if it encounters these status codes.                                                           | [500, 502, 503, 504, 522, 524, 408, 429]         |\n| itemConcurrency    | number   | The number of concurrent crawled item processing.                                                               | Same as request concurrency                      |\n| outputFormat       | string   | The output format such as json, csv, tsv.                                                                        | null                                             |\n| outputFileName     | string   | The output format name.                                                                                          | data                                             |\n| logLevel           | string   | The output log level.                                                                                            | debug                                            |\n| proxy              | object   | The proxy configuration.                                                                                         | null                                             |\n| proxyUrl           | string   | The proxy URL configuration.                                                                                     | null                                             |\n| continuous         | boolean  | For long-running Spidey processes such as listening to new URLs from Redis or Kafka.                             | false                                            |\n| pipelines          | SpideyPipeline[] | Item pipeline injection.                                                                                 | []                                               |\n\n\n## Data Pipeline\n\nSpidey enables the creation of personalized data pipelines for storing, validating, and manipulating the data that has been crawled.\n\n### Data Manipulation\n```typescript\nexport class ASINPipeline implements SpideyPipeline {\n  constructor(private options?: SpideyOptions) {}\n\n  process(data: Data) {\n    data.url = data.url.split('/ref').shift() as string;\n    data.asin = data.url.split('/').pop() as string;\n    return data;\n  }\n}\n```\n\n### Store Data to Mongodb\n```typescript\nexport class MongoPipeline implements SpideyPipeline {\n  client: MongoClient;\n  collection!: Collection;\n\n  constructor(private options?: SpideyOptions) {\n    this.client = new MongoClient(this.options?.mongoUrl);\n  }\n\n  async start() {\n    await this.client.connect();\n    const db = this.client.db(this.options?.mongoDb);\n    this.collection = db.collection(this.options?.mongoCollection);\n  }\n\n  async complete() {\n    await this.client.close();\n  }\n\n  async process(data: Data) {\n    await this.collection.findOneAndUpdate({ asin: data.asin }, { $set: data }, { upsert: true });\n\n    // Always return data back in pipelines in process function\n    return data;\n  }\n}\n```\n\n### Pipeline Usage\nPipelines can be injected to any spidey instance by passing in `pipelines` options.\n```typescript\n\nclass AmazonSpidey extends Spidey {\n  constructor() {\n    super({\n      // ...spidey options\n      pipelines: [ASINPipeline, MongoPipeline]\n    });\n  }\n}\n\n```\n\n## License\n\nSpidey is licensed under the [MIT License](https://opensource.org/licenses/MIT).\n","readmeFilename":"README.md"}