{"_id":"s3-parallel-get","_rev":"27-d2893453693afb832f418de279cc0b20","name":"s3-parallel-get","description":"Get files from s3, in parallel","dist-tags":{"latest":"0.2.2"},"versions":{"0.1.0":{"name":"s3-parallel-get","version":"0.1.0","author":{"name":"Matt Chadburn","email":"matt@commuterjoy.co.uk"},"description":"Get files from s3, in parallel","repository":{"type":"git","url":"https://github.com/commuterjoy/s3-parallel-get.git"},"keywords":["aws","s3"],"bin":{"s3get":"./bin/s3get"},"dependencies":{"aws-sdk":"*","moment":"*","async":"*","commander":"*"},"devDependencies":{"mocha":"*"},"license":"MIT","engines":{"node":">=0.10"},"bugs":{"url":"https://github.com/commuterjoy/s3-parallel-get/issues"},"homepage":"https://github.com/commuterjoy/s3-parallel-get","_id":"s3-parallel-get@0.1.0","dist":{"shasum":"955696adab03088ae6e386f7fe0acc423b97940f","tarball":"https://registry.npmjs.org/s3-parallel-get/-/s3-parallel-get-0.1.0.tgz","integrity":"sha512-1JEY7eUIBlFwbqDMulmTuAPF0+4ns21a288YQpWuBREghd8vhS8U9p2AFxMDQUxj4mVRJOJrHZv8iz1gefi8KA==","signatures":[{"keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA","sig":"MEYCIQDn36CsfLPU/rKKeYec8oU1wxADYxox+0yNHDzDiGNGGQIhAMiJCYwtd5WyyVHArJAg71OZIftD/7FV9S+3fJef7fQp"}]},"_from":".","_npmVersion":"1.3.15","_npmUser":{"name":"commuterjoy","email":"matt@commuterjoy.co.uk"},"maintainers":[{"name":"commuterjoy","email":"matt@commuterjoy.co.uk"}]},"0.1.1":{"name":"s3-parallel-get","version":"0.1.1","author":{"name":"Matt Chadburn","email":"matt@commuterjoy.co.uk"},"description":"Get files from s3, in parallel","repository":{"type":"git","url":"https://github.com/commuterjoy/s3-parallel-get.git"},"main":"lib/aws-parallel-get.js","preferGlobal":"true","keywords":["aws","s3"],"bin":{"s3get":"./bin/s3get"},"dependencies":{"aws-sdk":"2.*","moment":"2.5.*","async":"0.2.*","commander":"2.1.*"},"license":"MIT","engines":{"node":">=0.10"},"bugs":{"url":"https://github.com/commuterjoy/s3-parallel-get/issues"},"homepage":"https://github.com/commuterjoy/s3-parallel-get","_id":"s3-parallel-get@0.1.1","dist":{"shasum":"995b6bfd51f42de8b5ac1a91acc9001ebf9ca556","tarball":"https://registry.npmjs.org/s3-parallel-get/-/s3-parallel-get-0.1.1.tgz","integrity":"sha512-l1BTgVBhfTgp858N4t1GK4tGakzu3djmNr4ujSAxE21P8xKwx0ktPvsieM4dlwMIhQCvHzgb1/Xtzon2J9lqQg==","signatures":[{"keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA","sig":"MEUCIEYK36kg/4gpOT6DC5GM+okGMuOwss7JsTfOkjsfJpi4AiEA9z7nMOVLa3e3hfWHcLgokF3hqaxLzmnN1fLh5WseYvs="}]},"_from":".","_npmVersion":"1.3.15","_npmUser":{"name":"commuterjoy","email":"matt@commuterjoy.co.uk"},"maintainers":[{"name":"commuterjoy","email":"matt@commuterjoy.co.uk"}]},"0.1.2":{"name":"s3-parallel-get","version":"0.1.2","author":{"name":"Matt Chadburn","email":"matt@commuterjoy.co.uk"},"description":"Get files from s3, in parallel","repository":{"type":"git","url":"https://github.com/commuterjoy/s3-parallel-get.git"},"main":"lib/aws-parallel-get.js","preferGlobal":"true","keywords":["aws","s3"],"bin":{"s3get":"./bin/s3get"},"dependencies":{"aws-sdk":"2.*","moment":"2.5.*","async":"0.2.*","commander":"2.1.*"},"license":"MIT","engines":{"node":">=0.10"},"bugs":{"url":"https://github.com/commuterjoy/s3-parallel-get/issues"},"homepage":"https://github.com/commuterjoy/s3-parallel-get","_id":"s3-parallel-get@0.1.2","dist":{"shasum":"9f901c8a1ae494aa841ad2fae6d123e36ff395f9","tarball":"https://registry.npmjs.org/s3-parallel-get/-/s3-parallel-get-0.1.2.tgz","integrity":"sha512-s7cp5KSwUntMhBIAwPZlY/qTRU87xbLMsQcPsybLc9P3k+jqmAsTH6zZF2DyW3CtugRMZ9IzwPYTtpsznhxsrg==","signatures":[{"keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA","sig":"MEQCIC4C1qHv5ApqADXHcdLtQv2/2fUxhCLooshGBjTHqj0zAiAdP2EkaXf4OpDjmNQZqIB5uF5v4fH4XFhB2GLmtPH+gw=="}]},"_from":".","_npmVersion":"1.3.15","_npmUser":{"name":"commuterjoy","email":"matt@commuterjoy.co.uk"},"maintainers":[{"name":"commuterjoy","email":"matt@commuterjoy.co.uk"}]},"0.2.0":{"name":"s3-parallel-get","version":"0.2.0","author":{"name":"Matt Chadburn","email":"matt@commuterjoy.co.uk"},"description":"Get files from s3, in parallel","repository":{"type":"git","url":"https://github.com/commuterjoy/s3-parallel-get.git"},"main":"lib/aws-parallel-get.js","preferGlobal":"true","keywords":["aws","s3"],"bin":{"s3get":"./bin/s3get"},"dependencies":{"aws-sdk":"2.*","moment":"2.5.*","async":"0.2.*","commander":"2.1.*"},"license":"MIT","engines":{"node":">=0.10"},"bugs":{"url":"https://github.com/commuterjoy/s3-parallel-get/issues"},"homepage":"https://github.com/commuterjoy/s3-parallel-get","_id":"s3-parallel-get@0.2.0","dist":{"shasum":"91944ee7eaa396b6a404551cebd93b64c5016075","tarball":"https://registry.npmjs.org/s3-parallel-get/-/s3-parallel-get-0.2.0.tgz","integrity":"sha512-yLbB0b3/jHHjBy03gh/2hioJnhKGFXo02Hfj2AWqGf/yiP/0weCzD/Cax1a9QrBd96vrY+/RAInKPmEiTLhXsQ==","signatures":[{"keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA","sig":"MEQCIDoCdS3vPvYZnnMg6ybTKlp74TGOc1zb5aafQuIppeCfAiAZGWjH3yW6Wzz7SA+xaoxmBti7sj+La2VKbV35Lm5RJg=="}]},"_from":".","_npmVersion":"1.3.15","_npmUser":{"name":"commuterjoy","email":"matt@commuterjoy.co.uk"},"maintainers":[{"name":"commuterjoy","email":"matt@commuterjoy.co.uk"}]},"0.2.1":{"name":"s3-parallel-get","version":"0.2.1","author":{"name":"Matt Chadburn","email":"matt@commuterjoy.co.uk"},"description":"Get files from s3, in parallel","repository":{"type":"git","url":"https://github.com/commuterjoy/s3-parallel-get.git"},"main":"lib/aws-parallel-get.js","preferGlobal":"true","keywords":["aws","s3"],"bin":{"s3get":"./bin/s3get"},"dependencies":{"aws-sdk":"2.*","moment":"2.5.*","async":"0.2.*","commander":"2.1.*"},"license":"MIT","engines":{"node":">=0.10"},"bugs":{"url":"https://github.com/commuterjoy/s3-parallel-get/issues"},"homepage":"https://github.com/commuterjoy/s3-parallel-get","_id":"s3-parallel-get@0.2.1","dist":{"shasum":"352d69d3b31bf6ddd324f6225b4c286ccef0e7ad","tarball":"https://registry.npmjs.org/s3-parallel-get/-/s3-parallel-get-0.2.1.tgz","integrity":"sha512-O+xCOPJNgz1v6YxIQ3FXrNsBCkYFN3Du8202u2ubr+bwL//v4km2GYKPumrPaVBjEOYjneloadTQXhcOBO22/A==","signatures":[{"keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA","sig":"MEUCIQDmSJSFZJ3LQTzX8sN/sbC6GYCF8N70/XnoiM5p/tPPhwIgZyYdqFfVVXnoe2qwGe/ML60aepWtC+/vCCDQE4kQ624="}]},"_from":".","_npmVersion":"1.3.15","_npmUser":{"name":"commuterjoy","email":"matt@commuterjoy.co.uk"},"maintainers":[{"name":"commuterjoy","email":"matt@commuterjoy.co.uk"}]},"0.2.2":{"name":"s3-parallel-get","version":"0.2.2","author":{"name":"Matt Chadburn","email":"matt@commuterjoy.co.uk"},"description":"Get files from s3, in parallel","repository":{"type":"git","url":"https://github.com/commuterjoy/s3__parallel-get.git"},"main":"lib/aws-parallel-get.js","preferGlobal":"true","keywords":["aws","s3"],"bin":{"s3get":"./bin/s3get"},"dependencies":{"aws-sdk":"2.*","moment":"2.5.*","async":"0.2.*","commander":"2.1.*"},"license":"MIT","engines":{"node":">=0.10"},"bugs":{"url":"https://github.com/commuterjoy/s3__parallel-get/issues"},"homepage":"https://github.com/commuterjoy/s3__parallel-get","_id":"s3-parallel-get@0.2.2","dist":{"shasum":"1e4525c097bfc53d44886c404e74472c66e257f0","tarball":"https://registry.npmjs.org/s3-parallel-get/-/s3-parallel-get-0.2.2.tgz","integrity":"sha512-HfhgsBe+mX4RSt9yoemQw2hpkgdGVUKXVBBflCJN9j/jOFwWJI8FPuasi9pffI4Gb0EdfkQx4dKAV4CLYVhzZA==","signatures":[{"keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA","sig":"MEQCIHGoeNq7J9t9dTnPkgufKi/0mJ/v372RK9mZ/9Nqcq6TAiB0eoOFV6syove/k8uHJcFDeQTjGGTwAxkog82WN2uFfQ=="}]},"_from":".","_npmVersion":"1.3.15","_npmUser":{"name":"commuterjoy","email":"matt@commuterjoy.co.uk"},"maintainers":[{"name":"commuterjoy","email":"matt@commuterjoy.co.uk"}]}},"readme":"[s3cmd](http://s3tools.org/s3cmd) is great, but too slow for downloading lots of files from s3. Instead, **s3__parallel-get** lets you download many files concurrently, meaning you get your files quicker.\n\nDownloading 150MB of log files split over 150 files, **s3cmd** took around 100 seconds, whereas **s3-parallel-get** took just under 25 seconds.\n\n## Installation\n\nJust a standard npm install,\n\n```\nnpm install -g s3-parallel-get\n```\n\n## Usage\n\n```\nexport AWS_KEY=<key>\nexport AWS_SECRET=<secret>\n\ns3get --bucket <bucket> --prefix <path/to/files>\n```\n\n**s3__parallel-get** was built for fetching log files, so output is sent to `stdout` for piping in to other commands.\n\nProgress is printed to stderr,\n\n```\nfound 4 objects in 'logs/production/access.log/2014/02/22'\nretrieved object #2 - PROD/access.log/2014/02/22/access.i-13e5f250-b\nretrieved object #4 - PROD/access.log/2014/02/22/access.i-13e5f250-d\nretrieved object #3 - PROD/access.log/2014/02/22/access.i-13e5f250-c\nretrieved object #1 - PROD/access.log/2014/02/22/access.i-13e5f250-a\ndone\n```\n\n## Programmatic\n\n```\nvar s3get = require('../lib/aws-parallel-get').s3get\n\n// Set some options\nvar opts = {\n    bucket: program.bucket,\n    prefix: program.prefix,\n    key: program.key,\n    secret: program.secret\n});\n\nvar s3 = new s3get(opts)\n\n// write a handler for the s3 stream\ns3.on('data', function (data) {\n    process.stdout.write(\"*\" + data)\n})  \n\n// kick it off\ns3.go();\n```\n","maintainers":[{"name":"commuterjoy","email":"matt@commuterjoy.co.uk"}],"time":{"modified":"2022-06-26T15:11:55.553Z","created":"2014-02-24T21:25:20.581Z","0.1.0":"2014-02-24T21:25:22.659Z","0.1.1":"2014-02-24T21:35:16.799Z","0.1.2":"2014-02-25T07:45:30.109Z","0.2.0":"2014-02-25T07:48:34.118Z","0.2.1":"2014-02-25T07:53:13.675Z","0.2.2":"2014-02-25T07:58:12.781Z"},"readmeFilename":"README.md","homepage":"https://github.com/commuterjoy/s3__parallel-get","keywords":["aws","s3"],"repository":{"type":"git","url":"https://github.com/commuterjoy/s3__parallel-get.git"},"author":{"name":"Matt Chadburn","email":"matt@commuterjoy.co.uk"},"bugs":{"url":"https://github.com/commuterjoy/s3__parallel-get/issues"},"license":"MIT","users":{"shaner":true}}