{"_id":"@dshahi468/nd-dataset-split","_rev":"2-6c061b81377233c939c865abc861b443","name":"@dshahi468/nd-dataset-split","dist-tags":{"latest":"1.0.1"},"versions":{"1.0.0":{"name":"@dshahi468/nd-dataset-split","version":"1.0.0","keywords":["chunk","batch","dataset","promise-all","retry","concurrency","bedrock"],"author":{"name":"Dilendra Vikram Shahi"},"license":"ISC","_id":"@dshahi468/nd-dataset-split@1.0.0","maintainers":[{"name":"dshahi468","email":"dshahi468@gmail.com"}],"dist":{"shasum":"fd19a8e2237e919112e988d96f8fa6897ebab341","tarball":"https://registry.npmjs.org/@dshahi468/nd-dataset-split/-/nd-dataset-split-1.0.0.tgz","fileCount":4,"integrity":"sha512-iaZfVLaiZOW8se5TnuXnJHGZ7yWo/UMoqot6Mm01jgP3aI6/+D886dDWWK5ii5XdQYDBIjCh3oXkpoR+l0EkUw==","signatures":[{"sig":"MEQCIBS16RryvoLVRnpBwG88qW/haLC25S0X96VeBt2uDyBdAiBCxF/rDtl6WB97LVadsiV6LtRhnTlGuyK4hVIEX/9zDw==","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":11885},"main":"index.js","type":"module","types":"index.d.ts","engines":{"node":">=18"},"exports":{".":{"types":"./index.d.ts","import":"./index.js"}},"gitHead":"98210b0da7d8565f0ae6dddbcfcfdff78305e1a4","scripts":{"test":"echo \"Error: no test specified\" && exit 1"},"_npmUser":{"name":"dshahi468","email":"dshahi468@gmail.com"},"_npmVersion":"11.3.0","description":"Split large dataset and resolve timeout issues","directories":{},"_nodeVersion":"24.1.0","publishConfig":{"access":"public"},"_hasShrinkwrap":false,"_npmOperationalInternal":{"tmp":"tmp/nd-dataset-split_1.0.0_1772505586854_0.430560404764351","host":"s3://npm-registry-packages-npm-production"}},"1.0.1":{"name":"@dshahi468/nd-dataset-split","version":"1.0.1","description":"Split large dataset and resolve timeout issues","keywords":["chunk","batch","dataset","promise-all","retry","concurrency","bedrock"],"license":"ISC","author":{"name":"Dilendra Vikram Shahi"},"type":"module","main":"index.js","types":"index.d.ts","exports":{".":{"types":"./index.d.ts","import":"./index.js"}},"publishConfig":{"access":"public"},"engines":{"node":">=18"},"scripts":{"test":"echo \"Error: no test specified\" && exit 1"},"_id":"@dshahi468/nd-dataset-split@1.0.1","gitHead":"98210b0da7d8565f0ae6dddbcfcfdff78305e1a4","_nodeVersion":"24.1.0","_npmVersion":"11.3.0","dist":{"integrity":"sha512-saCIlulce+dqD7rif8pjRL/KUgz9PKDp1pspxu6pvtydjvn2vYIaV6ryKonSsOPW54eL1xyW/LqWLdrnOQhGgg==","shasum":"cf862bca93346bd43bf34fea45e712c99de5cbdf","tarball":"https://registry.npmjs.org/@dshahi468/nd-dataset-split/-/nd-dataset-split-1.0.1.tgz","fileCount":4,"unpackedSize":11901,"signatures":[{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEUCID4Mert8/SbuyMy2S1W7jPccf9tcZ1WeHZ4Lg8lE9dK3AiEAl4YrAxJR1/fQqypP1anNMaqd+4gmc+j8pw7b+GgZO8s="}]},"_npmUser":{"name":"dshahi468","email":"dshahi468@gmail.com"},"directories":{},"maintainers":[{"name":"dshahi468","email":"dshahi468@gmail.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/nd-dataset-split_1.0.1_1772505930595_0.8344239066192247"},"_hasShrinkwrap":false}},"time":{"created":"2026-03-03T02:39:46.750Z","modified":"2026-03-03T02:45:30.843Z","1.0.0":"2026-03-03T02:39:46.993Z","1.0.1":"2026-03-03T02:45:30.736Z"},"author":{"name":"Dilendra Vikram Shahi"},"license":"ISC","keywords":["chunk","batch","dataset","promise-all","retry","concurrency","bedrock"],"description":"Split large dataset and resolve timeout issues","maintainers":[{"name":"dshahi468","email":"dshahi468@gmail.com"}],"readme":"# nd-dataset-split\n\n大きなデータセットを安全に分割し、非同期処理をチャンク単位で実行するための軽量ユーティリティです。  \n「1リクエストあたり最大20件」などの制約があるAPI連携（例: Bedrock 呼び出し）を共通化できます。\n\n## 特徴\n\n- データを最大20件単位でチャンク化\n- `Promise.all` ベースの並列チャンク処理\n- 並列数（`maxConcurrency`）の制御\n- リトライ + バックオフ（指数バックオフ対応）\n- エラーごとの再試行可否を `shouldRetry` で制御\n- 結果のマージ処理を `mergeResults` で自由に定義\n- TypeScript 型定義同梱（`index.d.ts`）\n\n## インストール\n\n### npm レジストリから\n\n```bash\nnpm i @dshahi468/nd-dataset-split\n```\n\n## 使い方\n\n### 1. シンプルに使う（`processInChunks`）\n\n```ts\nimport { processInChunks } from \"@dshahi468/nd-dataset-split\";\n\nconst data = Array.from({ length: 45 }, (_, i) => i + 1);\n\nconst result = await processInChunks(\n  data,\n  async (chunk) => chunk.map((n) => n * 2),\n  { chunkSize: 20 }, // 1〜20\n);\n\nconsole.log(result.length); // 45\n```\n\n### 2. 実運用向け（`processDatasetInChunks`）\n\n```ts\nimport { processDatasetInChunks } from \"nd-dataset-split\";\n\nconst output = await processDatasetInChunks(\n  events,\n  async (chunk, chunkIndex, totalChunks) => {\n    // or you can use your logic here\n    const res = await fetch(\"your-api-name\", {\n      method: \"POST\",\n      headers: { \"Content-Type\": \"application/json\" },\n      body: JSON.stringify({\n        events: chunk,\n      }),\n    });\n\n    const payload = await res.json().catch(() => ({}));\n    if (!res.ok) {\n      throw new Error(\n        `Chunk ${chunkIndex + 1}/${totalChunks}: ${payload?.error || \"Failed\"}`,\n      );\n    }\n    return payload; // チャンク単位の結果\n  },\n  {\n    chunkSize: 20,\n    maxConcurrency: 3,\n    retries: 2,\n    retryDelayMs: 600,\n    backoffMultiplier: 1.5,\n    shouldRetry: (error) => {\n      const message = String((error as Error)?.message || \"\");\n      return message.includes(\"throttl\") || message.includes(\"timeout\");\n    },\n    mergeResults: (chunkResults) => chunkResults, // 必要に応じて独自マージ\n  },\n);\n```\n\n## API\n\n## `chunkArray(data, chunkSize = 20)`\n\n配列をチャンクに分割します。\n\n- `data`: 対象配列\n- `chunkSize`: 1〜20 の整数\n- 戻り値: `T[][]`\n\n## `wait(ms)`\n\n指定ミリ秒待機するヘルパーです。\n\n- `ms`: 待機時間（ミリ秒）\n- 戻り値: `Promise<void>`\n\n## `processInChunks(data, processChunk, options?)`\n\nシンプルなチャンク処理用APIです。  \n内部で全チャンクを `Promise.all` で一括実行します。\n\n- `data`: 対象配列\n- `processChunk`: `(chunk, chunkIndex, totalChunks) => Promise<R[] | R> | (R[] | R)`\n- `options.chunkSize`:\n  - デフォルト `20`\n  - 1〜20 の整数\n- `options.flatten`:\n  - デフォルト `true`\n  - `true`: 各チャンク結果を1つの配列にフラット化\n  - `false`: チャンクごとの結果配列をそのまま返却\n\n## `processDatasetInChunks(data, processChunk, options?)`\n\n実運用向けの拡張APIです。  \n並列数制御、リトライ、バックオフ、カスタムマージに対応します。\n\n- `data`: 対象配列\n- `processChunk`: `(chunk, chunkIndex, totalChunks) => Promise<R> | R`\n- `options.chunkSize`:\n  - デフォルト `20`\n  - 1〜20 の整数\n- `options.maxConcurrency`:\n  - デフォルト `Infinity`\n  - 同時実行するチャンク数\n- `options.retries`:\n  - デフォルト `0`\n  - 失敗時の再試行回数\n- `options.retryDelayMs`:\n  - デフォルト `0`\n  - リトライ間隔の基準ミリ秒\n- `options.backoffMultiplier`:\n  - デフォルト `1`\n  - 実遅延: `retryDelayMs * backoffMultiplier^attempt`\n- `options.shouldRetry`:\n  - デフォルト `() => true`\n  - `false` を返すと即時失敗\n- `options.mergeResults`:\n  - 指定時は `chunkResults` を任意の最終形式へ変換して返却\n  - 未指定時は `R[]`（チャンク結果配列）を返却\n\n## エラーハンドリングの考え方\n\n- API制限（429 / throttling）や一時的なタイムアウトは `shouldRetry` で再試行対象にする\n- 認可エラーや入力不正など恒久エラーは `shouldRetry` で `false` を返して即失敗\n- エラーメッセージに `chunkIndex + 1` と `totalChunks` を入れると追跡しやすい\n\n## 注意点\n\n- `chunkSize` は 20 を超えられません（仕様）\n- `maxConcurrency` を上げすぎると、相手API側のレート制限にかかりやすくなります\n- `mergeResults` で重複排除・集計・並び替えなどを実装すると、呼び出し側コードを薄くできます\n\n## ライセンス\n\nISC\n","readmeFilename":"README.md"}