{"_id":"@c-collamar/datasync-s3-transfer","_rev":"2-0f1b21d35514169f89f0ebeceb919361","name":"@c-collamar/datasync-s3-transfer","dist-tags":{"latest":"2.0.5"},"versions":{"2.0.2":{"name":"@c-collamar/datasync-s3-transfer","version":"2.0.2","description":"Automate S3 object transfers between buckets via DataSync","type":"module","main":"index.js","scripts":{"test":"echo \"Error: no test specified\" && exit 1","start":"node index.js"},"author":{"name":"Christian Collamar","email":"christian.collamar@gmail.com","url":"https://github.com/c-collamar"},"license":"BSD-3-Clause","dependencies":{"@aws-sdk/client-datasync":"^3.437.0","@aws-sdk/client-s3":"^3.435.0"},"imports":{"#lib/*.js":"./lib/*.js"},"exports":"./index.js","repository":{"type":"git","url":"git+https://github.com/C-Collamar/datasync-s3-transfer.git"},"_id":"@c-collamar/datasync-s3-transfer@2.0.2","gitHead":"2193d68b4882463899b2b281085e2d328515d54d","bugs":{"url":"https://github.com/C-Collamar/datasync-s3-transfer/issues"},"homepage":"https://github.com/C-Collamar/datasync-s3-transfer#readme","_nodeVersion":"20.10.0","_npmVersion":"10.2.3","dist":{"integrity":"sha512-psTCLheHUDg4rsSQE0zGH/hlmFUl572nO973yjT7Ezu7DiWD4zFoulXhyY8vOAcBED24uhMM1SwoF5++HF0C5w==","shasum":"c3aa95852518c7e3bc7144bd7540612f74017730","tarball":"https://registry.npmjs.org/@c-collamar/datasync-s3-transfer/-/datasync-s3-transfer-2.0.2.tgz","fileCount":7,"unpackedSize":33795,"attestations":{"url":"https://registry.npmjs.org/-/npm/v1/attestations/@c-collamar%2fdatasync-s3-transfer@2.0.2","provenance":{"predicateType":"https://slsa.dev/provenance/v1"}},"signatures":[{"keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA","sig":"MEUCICvWlwcEu7oOvIt4hO4OObjPTVALRgtiCMSbIedPbSMQAiEA1WCzTLzl90oZ1MHUZFLTyGFbjVNq+8Ps+ZuTvQgyifE="}]},"_npmUser":{"name":"c-collamar","email":"christian.collamar+npm@gmail.com"},"directories":{},"maintainers":[{"name":"c-collamar","email":"christian.collamar+npm@gmail.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages","tmp":"tmp/datasync-s3-transfer_2.0.2_1703787713518_0.20830667544659343"},"_hasShrinkwrap":false},"2.0.4":{"name":"@c-collamar/datasync-s3-transfer","version":"2.0.4","description":"Automate S3 object transfers between buckets via DataSync","type":"module","main":"index.js","scripts":{"test":"echo \"Error: no test specified\" && exit 1","start":"node index.js"},"author":{"name":"Christian Collamar","email":"christian.collamar@gmail.com","url":"https://github.com/c-collamar"},"license":"BSD-3-Clause","dependencies":{"@aws-sdk/client-datasync":"^3.437.0","@aws-sdk/client-s3":"^3.435.0"},"imports":{"#lib/*.js":"./lib/*.js"},"exports":"./index.js","repository":{"type":"git","url":"git+https://github.com/C-Collamar/datasync-s3-transfer.git"},"_id":"@c-collamar/datasync-s3-transfer@2.0.4","gitHead":"385ce4b3f328e457cbd8abfc91cdd765bebdd259","bugs":{"url":"https://github.com/C-Collamar/datasync-s3-transfer/issues"},"homepage":"https://github.com/C-Collamar/datasync-s3-transfer#readme","_nodeVersion":"20.10.0","_npmVersion":"10.2.3","dist":{"integrity":"sha512-iu4LvbmEXvDPXedXni82jMZZulgvZILWBNCcApiqO3UsVyQBZrIhQcqpaPhUv5yx8/SE24Ck5xxhFSaUuimgZg==","shasum":"9665168a1b96d2ea3da5b99302e4c0802d1e25a0","tarball":"https://registry.npmjs.org/@c-collamar/datasync-s3-transfer/-/datasync-s3-transfer-2.0.4.tgz","fileCount":8,"unpackedSize":34270,"attestations":{"url":"https://registry.npmjs.org/-/npm/v1/attestations/@c-collamar%2fdatasync-s3-transfer@2.0.4","provenance":{"predicateType":"https://slsa.dev/provenance/v1"}},"signatures":[{"keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA","sig":"MEUCIQCTQ9RnHZyqQgzgPV0ZuMKdrspBSBNIMX6p3PVmcIRLxgIgXQ7g4RL7YF1WDMBbCPXqcexAkkedA/vAR5Sw9TBgF00="}]},"_npmUser":{"name":"c-collamar","email":"christian.collamar+npm@gmail.com"},"directories":{},"maintainers":[{"name":"c-collamar","email":"christian.collamar+npm@gmail.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages","tmp":"tmp/datasync-s3-transfer_2.0.4_1703793006688_0.8972949191722175"},"_hasShrinkwrap":false},"2.0.5":{"name":"@c-collamar/datasync-s3-transfer","version":"2.0.5","description":"Automate S3 object transfers between buckets via DataSync","type":"module","main":"index.js","scripts":{"test":"echo \"Error: no test specified\" && exit 1","start":"node index.js"},"author":{"name":"Christian Collamar","email":"christian.collamar@gmail.com","url":"https://github.com/c-collamar"},"license":"BSD-3-Clause","dependencies":{"@aws-sdk/client-datasync":"^3.437.0","@aws-sdk/client-s3":"^3.435.0"},"imports":{"#lib/*.js":"./lib/*.js"},"exports":"./index.js","repository":{"type":"git","url":"git+https://github.com/C-Collamar/datasync-s3-transfer.git"},"keywords":["amazon","aws","datasync","s3"],"_id":"@c-collamar/datasync-s3-transfer@2.0.5","gitHead":"194571c41043e7a1020a69dda134f52aede48ae2","bugs":{"url":"https://github.com/C-Collamar/datasync-s3-transfer/issues"},"homepage":"https://github.com/C-Collamar/datasync-s3-transfer#readme","_nodeVersion":"20.10.0","_npmVersion":"10.2.3","dist":{"integrity":"sha512-aY6/y0tX71y+a4nGfQMudRMZnVB/6ouObj5KDYdxALN5gPm871gZPYBI3zu80FkjjTn0w7aDU3iNEnQ0sd4Nrw==","shasum":"0a9cc7837d5296db0cb6f0c6178b92ef0faabf03","tarball":"https://registry.npmjs.org/@c-collamar/datasync-s3-transfer/-/datasync-s3-transfer-2.0.5.tgz","fileCount":8,"unpackedSize":34345,"attestations":{"url":"https://registry.npmjs.org/-/npm/v1/attestations/@c-collamar%2fdatasync-s3-transfer@2.0.5","provenance":{"predicateType":"https://slsa.dev/provenance/v1"}},"signatures":[{"keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA","sig":"MEUCIQC8xb1QUbIqXWUDkrwS4lAwvOKqcBmSl/1MyIgdBCnGQgIgHCg7eXsI7WPBjN50rvIVcKL8Hy21lJYQF4+FUHX9C2s="}]},"_npmUser":{"name":"c-collamar","email":"christian.collamar+npm@gmail.com"},"directories":{},"maintainers":[{"name":"c-collamar","email":"christian.collamar+npm@gmail.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages","tmp":"tmp/datasync-s3-transfer_2.0.5_1703916723998_0.5535903899236099"},"_hasShrinkwrap":false}},"time":{"created":"2023-12-28T18:21:53.407Z","2.0.2":"2023-12-28T18:21:53.691Z","modified":"2023-12-30T06:12:04.659Z","2.0.4":"2023-12-28T19:50:06.898Z","2.0.5":"2023-12-30T06:12:04.151Z"},"maintainers":[{"name":"c-collamar","email":"christian.collamar+npm@gmail.com"}],"description":"Automate S3 object transfers between buckets via DataSync","homepage":"https://github.com/C-Collamar/datasync-s3-transfer#readme","repository":{"type":"git","url":"git+https://github.com/C-Collamar/datasync-s3-transfer.git"},"author":{"name":"Christian Collamar","email":"christian.collamar@gmail.com","url":"https://github.com/c-collamar"},"bugs":{"url":"https://github.com/C-Collamar/datasync-s3-transfer/issues"},"license":"BSD-3-Clause","readme":"# DataSync S3 Transfer\nAutomate S3 object transfers between buckets via DataSync!\n\n## Use Case\n\n- Transfer S3 objects between AWS S3 buckets via DataSync, without having to click around the AWS Console.\n- The necessary DataSync resources are generated for you, so you you don't have to.\n- S3 transfer can be between buckets in the same AWS accounts, or across different accounts.\n- for cross-account transfers, you can specify whether the source or destination AWS account does the transfer. (See issue [#1][3].)\n- Create automation scripts using this tool, to initiate batch S3 object transfers between many buckets.\n\n## Usage\n\n1. Install this package.\n\n   ```bash\n   npm install @c-collamar/datasync-s3-transfer\n   ```\n\n1. Prepare inputs.\n\n   ```js\n   // source aws profile to the aws account where your source bucket is in\n   const srcAwsProfile = {\n     region: 'ap-northeast-1',\n     credentials: {\n       accessKeyId: 'ABCDEFGHIJKLMNOPQRST',\n       secretAccessKey: 'WhHGQwvmvDaTne9LnMHV72A4cUkPkZWv2q6ieFtX'\n     }\n   };\n   \n   // if source and destination buckets belong to the same aws account, then...\n   const destAwsProfile = srcAwsProfile;\n   \n   const dataSyncOptions = {\n     /**\n      * Whether it is the \"source\" or \"destination\" AWS account that does the\n      * copying of S3 objects, i.e., the initiating account. For cross-account\n      * transfers, this dictates which AWS account the necessary DataSync\n      * resources will be created under. But for same-account transfers, then\n      * setting we want is...\n      */\n     initiatingAccount: 'source',\n    \n     // source or destination aws account id, whichever is the initiating account\n     dataSyncPrincipal: '123456789012',\n\n     /**\n      * Exisging IAM role from the initiating AWS account, with access to the\n      * source and destination buckets. (See notes on permissions.)\n      */\n     dataSyncRole: 'arn:aws:iam::123456789012:role/MyExistingRole',\n   \n     // existing cloudwatch log group under the initiating account, where datasync will record logs into\n     cloudWatchLogGroup: 'arn:aws:logs:ap-northeast-1:123456789012:log-group:/aws/datasync:*'\n   };\n   ```\n\n   Notes:\n\n   - If you want to do cross-account S3 object transfers, i.e., where the destination bucket is owned by a different AWS account than the source bucket, then set a different profile for `destAwsProfile` similar to `srcAwsProfile`.\n   - the IAM user behind the AWS config provided in `srcAwsProfile` or `destAwsProfile`, depending if `initiatingAccount` equals `source` or `destination` respectively, must have DataSync-related permissoins set up first. See requirements on [IAM User Permissions](#iam-user-permissions).\n   - The IAM role specified in `dataSyncoptions.dataSyncRole` must have the necessary permissions for DataSync to assume. See [IAM Role Permissions](#iam-role-permissions) on how the role must be setup.\n\n1. Start the transfer. That's it!\n   ```js\n   import { initDataSyncS3Transfer } from \"@c-collamar/datasync-s3-transfer\";\n\n   // configure transfer settings\n   const transfer = initDataSyncS3Transfer(srcAwsConfig, destAwsConfig, dataSyncOptions);\n   \n   // start the transfer\n   await transfer(\n     'source-bucket',              // existing source bucket name\n     'destination-bucket',         // existing destination bucket name\n     'Transfer to my other bucket' // the name of this transfer\n   );\n   ```\n\n   After the `transfer()` call is made, source S3 objects will be copied to the destination bucket the after some time. See under the hood on [How A Transfer Is Made](#how-a-transfer-is-made).\n\n1. Multiple transfers? Yes we can!\n\n   ```js\n   // initialize once\n   const transfer = initDataSyncS3Transfer(srcAwsConfig, destAwsConfig, dataSyncOptions);\n   \n   const toTransfer = [\n     { from: 'source-bucket-1', to: 'destination-bucket-1', name: 'Transfer task 1' },\n     { from: 'source-bucket-2', to: 'destination-bucket-2', name: 'Transfer task 2' },\n     // ...etc\n   ];\n   \n   // transfer multiple times\n   for (const item of toTransfer) {\n     await transfer(item.from, item.to, item.name);\n   }\n   \n   // or if you want to initiate the transfers concurrently...\n   await Promise.all(\n     toTransfer.map((item) => transfer(item.from, item.to, item.name))\n   );\n   ```\n\n1. How to properly handle errors? See [Error Handling and Retries](#error-handling-and-retries) for details.\n\n## How a Transfer is Made\n\nEverytime a transfer call is made, the following happens under the hood within the initiating AWS account:\n\n1. A DataSync source S3 location is created, poining to the source S3 bucket.\n\n   If the initiating AWS account does not own the source bucket, then the source bucket policy is first updated in order to permit the initiating account to create the DataSync location.\n\n1. A DataSync destination S3 location is created, pointing to the destination S3 bucket.\n\n   If the initiating AWS account does not own the destination bucket, then the destination bucket policy is first updated in order to permit the initiating account to create the DataSync location. \n\n1. A DataSync transfer task is created.\n\n1. The DataSync task is then executed, which creates a DataSync task execution resource and initiates the transfer of S3 objects between the source and destination DataSync locations.\n\nInformation on these created resources are then returned from the transfer call, into the `result` variable as seen below.\n\n```js\n// start the transfer\nconst { result, error } = await transfer('source-bucket', 'destination-bucket', 'Transfer to my other bucket');\n```\n\n## Error Handling and Retries\n\nErrors can happen at any point during the transfer process, from network issues all the way up to misconfigurations on your part. In any case, you can check for errors via the `error` property returned by the transfer call.\n\n```js\nconst { result, error } = await transfer('source-bucket', 'destination-bucket', 'Transfer to my other bucket');\n\nif (error) {\n  console.error(error);\n}\nelse {\n  console.info('Transfer successful!');\n}\n```\n\n### Retrying a Failed Transfer\n\nUnderstanding [How a Transfer is Made](#how-a-transfer-is-made), it is possible for an unexpected error (e.g., network or system error) to arise during any of the AWS resource-creation step. This can cause the transfer to become incomplete, where some of the necessary AWS resources have already been created, while the rest have not yet.\n\nIn this case, you can retry the transfer as follows:\n\n```js\nlet transferState = await transfer('source-bucket', 'destination-bucket', 'Transfer to my other bucket');\n\n// retry on error\nwhile (transferState.error) {\n  console.error(transferState.error);\n  console.info('Retrying...');\n\n  // you can also make retry optional by prompting retry confirmation first before executing this retry statement below\n  transferState = await transfer('source-bucket', 'destination-bucket', 'Transfer to my other bucket', transferState.result);\n}\n\n// you can also retry despite script termination by exporting transferState.result, to a file for example\n```\n\nThis way, successfully created AWS resources will be reused when retrying the transfer. Without supplying the `transferState.result` argument in the example above, calling `transfer()` mutiple times will create a new set of resources, which will likely cause AWS to complain about resource duplication.\n\n## Permissions\n\n### IAM User Permissions\n\nIn order to create the necessary DataSync resources on the initiating AWS account, this script assumes the IAM user behind the provided AWS config of the initiating account. In other words, using the code snippet from the [Usage](#usage) section as context, if `initiatingAccount` equals `source`, then the `srcAwsProfile` is used to create said resources. Else, if `initiatingAccount` equals `destination`, then the `destAwsProfile` is used instead.\n\nThis implies that the IAM user assumed by this script must be permitted certain actions in order to setup the DataSync-S3 transfer. These actions are:\n\n- `datasync:CancelTaskExecution`\n- `datasync:CreateLocationS3`\n- `datasync:CreateTask`\n- `datasync:DescribeLocation*`\n- `datasync:DescribeTask`\n- `datasync:DescribeTaskExecution`\n- `datasync:ListLocations`\n- `datasync:ListTasks`\n- `datasync:ListTaskExecutions`\n- `datasync:StartTaskExecution`\n- `iam:AttachRolePolicy`\n- `iam:CreateRole`\n- `iam:CreatePolicy`\n- `iam:ListRoles`\n- `iam:PassRole`\n- `s3:GetBucketLocation`\n- `s3:ListAllMyBuckets`\n- `s3:ListBucket`\n\n### IAM Role Permissions\n\nThe IAM role, whose ARN is passed in the `srcDataSyncRole` DataSync initialization option, must have the following permission setup as mentioned in [Creating an IAM role for DataSync to access your S3 bucket][4].\n\n**Role trust relationship:**\n\n```json\n{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": [\n    {\n      \"Effect\": \"Allow\",\n      \"Principal\": {\n        \"Service\": \"datasync.amazonaws.com\"\n      },\n      \"Action\": \"sts:AssumeRole\",\n      \"Condition\": {\n        \"StringEquals\": {\n          \"aws:SourceAccount\": \"ACCOUNT_ID\"\n        },\n        \"StringLike\": {\n          \"aws:SourceArn\": \"arn:aws:datasync:AWS_REGION:ACCOUNT_ID:*\"\n        }\n      }\n    }\n  ]\n}\n```\n\nReplace `ACCOUNT_ID` with the AWS account ID that owns the role, and `AWS_REGION` with the region you only want DataSync to work on. Restricting the trust relationship this way is recommended to prevent the cross-service deputy problem.\n\n**Role permission policy:**\n\n```json\n{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": [\n    {\n      \"Action\": [\n        \"s3:GetBucketLocation\",\n        \"s3:ListBucket\",\n        \"s3:ListBucketMultipartUploads\"\n      ],\n      \"Effect\": \"Allow\",\n      \"Resource\": \"arn:aws:s3:::*\"\n    },\n    {\n      \"Action\": [\n        \"s3:AbortMultipartUpload\",\n        \"s3:DeleteObject\",\n        \"s3:GetObject\",\n        \"s3:ListMultipartUploadParts\",\n        \"s3:GetObjectTagging\",\n        \"s3:PutObjectTagging\",\n        \"s3:PutObject\"\n      ],\n      \"Effect\": \"Allow\",\n      \"Resource\": \"arn:aws:s3:::*/*\"\n    }\n  ]\n}\n```\n\nFor the role permission policy, the idea is to allow certain actions on S3 objects that belong to the source and destination buckets. And by scoping the `Resource` properties as such, you can initiate S3 object transfers from _any_ buckets in the source AWS accuont, to _any_ buckets in the destination AWS account.\n\nThat being said, you can limit the scope of the `Resource` policy elements by explicitly listing the source and destination buckets only. Just keep in mind that S3 object transfers will not work if neither the source nor destination bucket is not included in this scope.\n\n## High-Level System Design\n\n```mermaid\nflowchart TD\n    start(((Start)))\n    in[/Source and destination buckets,\\nAWS creds, DataSync options, etc./]\n    policy[[Update source or destination\\nbucket policy as necessary]]\n    locs[[Create DataSync source and\\ndestination S3 locations]]\n    task[[Create DataSync\\ntransfer task]]\n    exec[[Execute\\ntransfer task]]\n    out[/DataSync source S3 location,\\nDataSync destination S3 location,\\nDataSync task,\\nDataSync task execution/]\n    stop(((Stop)))\n\n    start -- input --> in\n    in --> policy\n    policy --> locs\n    locs --> task\n    task --> exec\n    exec -- output --> out\n    out --> stop\n```\n\n## License\n\nThis project is licensed under the terms of the the 3-Clause BSD license.\n\n[2]: https://repost.aws/knowledge-center/s3-large-transfer-between-buckets\n[3]: https://github.com/C-Collamar/datasync-s3-transfer/issues/1\n[4]: https://docs.aws.amazon.com/datasync/latest/userguide/create-s3-location.html#create-role-manually","readmeFilename":"readme.md","keywords":["amazon","aws","datasync","s3"]}