{"_id":"@abap/sitemap-crawler","name":"@abap/sitemap-crawler","dist-tags":{"latest":"1.1.0"},"versions":{"1.1.0":{"name":"@abap/sitemap-crawler","version":"1.1.0","description":"Crawl links from a sitemap.xml file and extract the link to binary files.","type":"module","homepage":"https://github.com/tobiashofmann/sitemap-crawler","bugs":{"url":"https://github.com/tobiashofmann/sitemap-crawler/issues"},"repository":{"type":"git","url":"git+https://github.com/tobiashofmann/sitemap-crawler.git"},"bin":{"sitemap-links":"index.js"},"keywords":["sitemap","crawler"],"main":"index.js","scripts":{"start":"node index.js","lint":"eslint *.js"},"engines":{"node":"^20.11.0 || >=22.0.0","npm":">= 8"},"author":{"name":"Tobias Hofmann"},"license":"Apache-2.0","dependencies":{"fast-xml-parser":"^3.16.0","node-fetch":"^2.6.0"},"devDependencies":{"@eslint/js":"^9.35.0","eslint":"^9.35.0","globals":"^16.4.0"},"gitHead":"1b375415254bfb834e34888a3acad94dc5909f91","_id":"@abap/sitemap-crawler@1.1.0","_nodeVersion":"24.12.0","_npmVersion":"11.7.0","dist":{"integrity":"sha512-7AMTXg7/2Sc4dqlKFK+9zdr+JmTktRwYO7GpSwSoYRIV7RPDGkPMlR77NzViXz0uRcM3hn7P8dlvhOlfw1OvSg==","shasum":"53e8ed0aeefbc78147e7a43ddad93cbc6c390b81","tarball":"https://registry.npmjs.org/@abap/sitemap-crawler/-/sitemap-crawler-1.1.0.tgz","fileCount":8,"unpackedSize":22317,"signatures":[{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEQCIQC5eZxNqgP2ZAR+Cd7IL1FftscKuJ6mIwbWINcFKVdgSAIfHE/+MAkNz1pXyRJvHJ5fiQXby6v3A90wmgWMnC8JpA=="}]},"_npmUser":{"name":"tobiashofmann","email":"hofmann.tobias@outlook.com"},"directories":{},"maintainers":[{"name":"tobiashofmann","email":"hofmann.tobias@outlook.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/sitemap-crawler_1.1.0_1768402547088_0.6898091536360083"},"_hasShrinkwrap":false}},"time":{"created":"2026-01-14T14:55:46.952Z","1.1.0":"2026-01-14T14:55:47.239Z","modified":"2026-01-14T14:55:47.609Z"},"maintainers":[{"name":"tobiashofmann","email":"hofmann.tobias@outlook.com"}],"description":"Crawl links from a sitemap.xml file and extract the link to binary files.","homepage":"https://github.com/tobiashofmann/sitemap-crawler","keywords":["sitemap","crawler"],"repository":{"type":"git","url":"git+https://github.com/tobiashofmann/sitemap-crawler.git"},"author":{"name":"Tobias Hofmann"},"bugs":{"url":"https://github.com/tobiashofmann/sitemap-crawler/issues"},"license":"Apache-2.0","readme":"# sitemap-crawler\n\nCrawl links from a sitemap.xml file and extract the link to binary files.\n\n## Run\n\nClone the repo, install the dependencies and provide a sitemap URL.\n\n```sh\nnpm i\nnpm start [[sitemap url]]\n```\n\nThe URL to a sitemap can be provided as variable in file index.js:\n\n```javascript\nconst configUrl = \"<path to sitemap.xml>\";\n```\n\n## Result\n\nThe provided sitemap will be crawled recursively for links. The file data.json is a JSON array with all links found. The file links.txt contains all links to binary files like PDF, PPTX, DOCX, etc.\n\nThe file links.txt can be used as input for wget to download the files.\n\n## Example: download files using wget for Windows\n\nMake sure you have wget for Windows installed. You can get it here: https://eternallybored.org/misc/wget/\n\nDownload e.g. the exe version and save it in the directory of this project. Provide as input the links.txt file to download all files.\n\n```sh\n .\\wget.exe --content-disposition -i .\\links.txt\n```\n\n**Note**\n\nMake sure to only download the files you need! Depending on the sitemap.xml, there might be none or thousands of files!\n\n## Example\n\nAs an example, take the sitemap.xml from SAP Help. The location of the sitemap is given by the file [robots.txt](https://help.sap.com/robots.txt). At the end of the file: \n\nSitemap: https://help.sap.com/http.svc/sitemapxml/sitemaps/sitemap_index.xml\n\nThe URL can be used as input for the script\n\nnpm run https://help.sap.com/http.svc/sitemapxml/sitemaps/sitemap_index.xml\n\n### Output\n\n```sh\nnpm start\n\n> sitemap-crawler@1.0.0 start\n> node index.js\n\nStarting to crawl\nCrawling sitemap: https://help.sap.com/http.svc/sitemapxml/sitemaps/sitemap_index.xml\nFound another sitemap\nFound 60131 links in sitemap\ncrawling done.\nFile: data.json contains all links retrieved from sitemap.\nFile: links.txt contains all links to binary files (PDF, DOCX, PPTX, etc.) retrieved from sitemap.\n```\n\nThe file data.json will contain an array with over 60.000 entries. The file links.txt will list over 22.500 entries. To download those rung\n\n```sh\n .\\wget.exe --content-disposition -i .\\links.txt\n```\n\n**again:** you might not want to download all the files. Make sure to clean the list to what you really need!","readmeFilename":"README.md","_rev":"1-0e3dcf4af82d5494d3a04ac1da081f90"}