{"_id":"k-batch","_rev":"2-f91942431875ed8997de644e0444d3bf","name":"k-batch","dist-tags":{"latest":"1.0.1"},"versions":{"1.0.0":{"name":"k-batch","version":"1.0.0","keywords":["text","batching","k-means","clustering","optimal processing"],"author":"","license":"ISC","_id":"k-batch@1.0.0","maintainers":[{"name":"jparkerweb","email":"jparkerweb@gmail.com"}],"homepage":"https://github.com/jparkerweb/k-batch#readme","bugs":{"url":"https://github.com/jparkerweb/k-batch/issues"},"dist":{"shasum":"22fb0a17a8f85ab76f7ce7fea1ff99a404446fe2","tarball":"https://registry.npmjs.org/k-batch/-/k-batch-1.0.0.tgz","fileCount":4,"integrity":"sha512-y6r3BT0Fi3YcE1n3eBxeF5/bLawRsqAYkqzTuSPPPcFgxtNFTOp20TsludE1dF02lLmMV/RvZmwuh3/d0tI3XA==","signatures":[{"sig":"MEUCIQDVT5sBLgGcoFnfkSk5GvcIIJWSioRVtN+9L+SfVBx//AIgIA/iiWSsiVGZUQo64GDDzRrdvGqe4/jFD6PNMhbB0F0=","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":14566},"main":"k-batch.js","type":"module","funding":{"url":"https://ko-fi.com/jparkerweb","type":"ko-fi"},"gitHead":"7c7c3eba2af8c5fb7c312ea91a91975fa4a43dca","scripts":{"test":"node test.js"},"_npmUser":{"name":"jparkerweb","email":"jparkerweb@gmail.com"},"repository":{"url":"git+https://github.com/jparkerweb/k-batch.git","type":"git"},"_npmVersion":"10.2.4","description":"Intelligent text batching library that uses k-means clustering to group sentences by length for optimal processing.","directories":{},"_nodeVersion":"22.14.0","_hasShrinkwrap":false,"_npmOperationalInternal":{"tmp":"tmp/k-batch_1.0.0_1741900446666_0.5616131455710465","host":"s3://npm-registry-packages-npm-production"}},"1.0.1":{"name":"k-batch","version":"1.0.1","description":"Intelligent text batching library that uses k-means clustering to group sentences by length for optimal processing.","type":"module","main":"k-batch.js","scripts":{"test":"node test.js"},"funding":{"type":"ko-fi","url":"https://ko-fi.com/jparkerweb"},"keywords":["text","batching","k-means","clustering","optimal processing"],"author":"","license":"ISC","repository":{"type":"git","url":"git+https://github.com/jparkerweb/k-batch.git"},"bugs":{"url":"https://github.com/jparkerweb/k-batch/issues"},"_id":"k-batch@1.0.1","gitHead":"75a835fd76bbba17751c4c76bbf2671ff07d21e3","homepage":"https://github.com/jparkerweb/k-batch#readme","_nodeVersion":"22.14.0","_npmVersion":"10.2.4","dist":{"integrity":"sha512-b4x2hcHN0oP2Jg34gPuu4+MeJQRqpw8IPLVwPEPk6N91pH0AhXzrtGpdXT2xGeM1ZyjYeEUiEMBLfLPqRMtygg==","shasum":"ea52569e3dfe451e1ff92442a01d2e38123c0764","tarball":"https://registry.npmjs.org/k-batch/-/k-batch-1.0.1.tgz","fileCount":4,"unpackedSize":15032,"signatures":[{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEUCIEkksOFI98HYcKQVlAndsSEeTI3aCwvqnYIzGQW45dK5AiEA10kbXtZgvZpaGevqhUqfJleyoo5XDqCujqNj7nnsv9U="}]},"_npmUser":{"name":"jparkerweb","email":"jparkerweb@gmail.com"},"directories":{},"maintainers":[{"name":"jparkerweb","email":"jparkerweb@gmail.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/k-batch_1.0.1_1742265528355_0.3525228283986379"},"_hasShrinkwrap":false}},"time":{"created":"2025-03-13T21:14:06.665Z","modified":"2025-03-18T02:38:48.681Z","1.0.0":"2025-03-13T21:14:06.830Z","1.0.1":"2025-03-18T02:38:48.527Z"},"bugs":{"url":"https://github.com/jparkerweb/k-batch/issues"},"license":"ISC","homepage":"https://github.com/jparkerweb/k-batch#readme","keywords":["text","batching","k-means","clustering","optimal processing"],"repository":{"type":"git","url":"git+https://github.com/jparkerweb/k-batch.git"},"description":"Intelligent text batching library that uses k-means clustering to group sentences by length for optimal processing.","maintainers":[{"name":"jparkerweb","email":"jparkerweb@gmail.com"}],"readme":"# K-Batch\r\n\r\nK-Batch is an intelligent text batching library that uses k-means clustering to group sentences by length for optimal processing. It's particularly useful for NLP tasks, machine learning batch processing, and any scenario where processing similar-length texts together improves efficiency.\r\n\r\n![k-batch](https://raw.githubusercontent.com/jparkerweb/k-batch/refs/heads/main/.readme/k-batch.jpg)\r\n\r\n#### Maintained by\r\n<a href=\"https://www.equilllabs.com\">\r\n  <img src=\"https://raw.githubusercontent.com/jparkerweb/eQuill-Labs/refs/heads/main/src/static/images/logo-text-outline.png\" alt=\"eQuill Labs\" height=\"32\">\r\n</a>\r\n\r\n## Why K-Batch?\r\n\r\nWhen processing text in batches (especially for machine learning or NLP tasks), grouping sentences of similar length together can:\r\n\r\n- Reduce padding waste\r\n- Improve computational efficiency\r\n- Optimize memory usage\r\n- Enhance model training performance\r\n\r\nK-Batch uses k-means clustering to automatically group your sentences into optimal batches based on length, while ensuring each batch meets minimum size requirements.\r\n\r\n## Installation\r\n\r\n```bash\r\nnpm install k-batch\r\n```\r\n\r\n## Quick Start\r\n\r\n```javascript\r\nimport { kBatchSentences, analyzeKBatches } from 'k-batch';\r\n\r\nconst sentences = [\r\n  \"This is a short sentence.\",\r\n  \"A significantly longer sentence that should be in a different batch.\",\r\n  \"Tiny.\",\r\n  \"Here is another medium-length sentence.\",\r\n  \"One more sentence to make it interesting.\",\r\n  \"And another one to round out the collection.\",\r\n  \"Make it interesting.\",\r\n  \"And another one to round out the collection.\",\r\n  \"wow, this is short.\",\r\n  \"Who?\",\r\n  // ... more sentences\r\n];\r\n\r\n// Get optimally batched sentences\r\nconst batches = await kBatchSentences(sentences);\r\n\r\n// Use your batches\r\nbatches.forEach((batch, index) => {\r\n  console.log(`Batch ${index + 1}: ${batch.length} sentences`);\r\n  console.log(batch);\r\n  // Process each batch...\r\n});\r\n\r\n// Get detailed statistics about your batches\r\nconst stats = await analyzeKBatches(batches);\r\nconsole.log(stats);\r\n```\r\n\r\n## API Reference\r\n\r\n### kBatchSentences(sentences, options)\r\n\r\nThe main function that batches sentences using k-means clustering.\r\n\r\n#### Parameters\r\n\r\n- `sentences` (Array): Array of strings to be batched\r\n- `options` (Object, optional): Configuration options\r\n  - `maxBatches` (Number): Maximum number of batches to create (default: 5)\r\n  - `minSentencesPerBatch` (Number): Minimum sentences per batch (default: 4)\r\n  - `minSentencesRequired` (Number): Minimum number of sentences required to perform splitting (default: 10)\r\n  - `maxIterations` (Number): Maximum k-means iterations (default: 100)\r\n\r\n#### Returns\r\n\r\n- Array of arrays, where each inner array contains batched sentences\r\n\r\n## Advanced Usage\r\n\r\n### Custom Configuration\r\n\r\n```javascript\r\nimport { kBatchSentences } from 'k-batch';\r\n\r\nconst sentences = [/* your sentences */];\r\n\r\nconst batches = await kBatchSentences(sentences, {\r\n  maxBatches: 3,\r\n  minSentencesPerBatch: 5,\r\n  minSentencesRequired: 15,\r\n  maxIterations: 50\r\n});\r\n```\r\n\r\n### Analyzing Batch Statistics\r\n\r\n```javascript\r\nimport { kBatchSentences, analyzeKBatches } from 'k-batch';\r\n\r\nconst sentences = [/* your sentences */];\r\nconst batches = await kBatchSentences(sentences);\r\n\r\n// Get detailed statistics about your batches\r\nconst stats = await analyzeKBatches(batches);\r\nconsole.log(stats);\r\n/* Output:\r\n[\r\n  {\r\n    count: 11,\r\n    longestLength: 39,\r\n    shortestLength: 5,\r\n    averageLength: 24.09,\r\n    standardDeviation: 9.87\r\n  },\r\n  // ... stats for other batches\r\n]\r\n*/\r\n```\r\n\r\n## How It Works\r\n\r\nK-Batch uses a modified k-means clustering algorithm to group sentences by length:\r\n\r\n1. **Initial Clustering**: Sentences are clustered based on their character length using k-means\r\n2. **Batch Enforcement**: Small clusters are merged to ensure each batch meets the minimum size requirement\r\n3. **Sorting**: Within each batch, sentences are sorted from longest to shortest for optimal processing\r\n\r\nThe algorithm automatically determines the optimal number of clusters based on your data and constraints.\r\n\r\n## Use Cases\r\n\r\n- **NLP Processing**: Group similar-length texts for more efficient tokenization and embedding\r\n- **Machine Learning**: Create optimized batches for training language models\r\n- **API Requests**: Batch similar-length texts together when making API calls to language models\r\n- **Text Generation**: Process prompts of similar lengths together for better throughput\r\n\r\n## Performance Considerations\r\n\r\n- K-Batch is optimized for datasets with varying sentence lengths\r\n- For very large datasets (>10,000 sentences), consider processing in chunks\r\n- The algorithm's time complexity is O(n * k * i), where:\r\n  - n = number of sentences\r\n  - k = number of clusters\r\n  - i = number of iterations\r\n\r\n## Web UI\r\n\r\nK-Batch includes a simple web interface to help you visualize and experiment with the batching algorithm. The Web UI allows you to:\r\n\r\n- Input your own text and see how it gets batched in real-time\r\n- Adjust parameters using interactive sliders\r\n- View detailed statistics for each batch\r\n- Experiment with different text patterns to understand the clustering behavior\r\n\r\nTo use the Web UI:\r\n\r\n```bash\r\ncd webui\r\nnpm install\r\nnpm start\r\n```\r\n\r\nThis will start a local server and open the interface in your browser. For more details, see the [Web UI README](webui/README.md).\r\n\r\n#### 🌐 Online Demo of the Web UI can be found here: https://k-batch.dyndns.org/\r\n\r\n## Contributing\r\n\r\nContributions are welcome! Please feel free to submit a Pull Request.\r\n\r\n1. Fork the repository\r\n2. Create your feature branch (`git checkout -b feature/amazing-feature`)\r\n3. Commit your changes (`git commit -m 'Add some amazing feature'`)\r\n4. Push to the branch (`git push origin feature/amazing-feature`)\r\n5. Open a Pull Request\r\n\r\n## License\r\n\r\nThis project is licensed under the MIT License - see the LICENSE file for details.\r\n\r\n## Acknowledgments\r\n\r\n- Inspired by the need for efficient text processing in NLP tasks\r\n- Uses a modified k-means algorithm optimized for text length clustering\r\n\r\n---\r\n\r\nMade with ❤️ for the NLP and ML community\r\n","readmeFilename":"README.md"}