![\"CRYSTAL](\"https://raw.githubusercontent.com/AndrewShedov/mongoCollector/refs/heads/main/assets/screenshot_2.1.png\")
\n\nfield from MongoDB documents.\n2. Data filtering using $match.\n3. Batching (batchSize) to avoid MongoDB’s 16MB per-document limit.\n4. ObjectId transformation: ObjectId('68a8c8207090be6dd0e23a90') → '68a8c8207090be6dd0e23a90'.\n5. Large collections supported via allowDiskUse.\n6. Flexible array handling:\n - Overwrite or append to arrays.\n - Allow or eliminate duplicates.\n7. Informative logs:\n![](\"https://raw.githubusercontent.com/AndrewShedov/mongoCollector/refs/heads/main/assets/screenshot_1.png\")
\n\n### Installation & Usage\n\n1. Install the package:\n\n```bash\nnpm i mongo-collector\n```\n\n2. Add a script in your **package.json**:\n\n```json\n\"scripts\": {\n \"mongoCollector\": \"mongo-collector\"\n}\n```\n\n3. In the root of the project, create a file — [mongo-collector.config.js](https://github.com/AndrewShedov/mongoCollector/blob/main/config%20example/mongo-collector.config.js).\n\nExample of file contents:\n\n```js\nexport default {\n source: {\n uri: \"mongodb://127.0.0.1:27017\",\n db: \"crystalTest\",\n collection: \"users\",\n field: \"_id\",\n match: {}\n },\n\n target: {\n uri: \"mongodb://127.0.0.1:27017\",\n db: \"pool\",\n collection: \"usersIdFromCrystalTest\",\n field: \"users\",\n documentId: false,\n rewriteDocuments: true,\n rewriteArray: true,\n duplicatesInArray: false,\n unwrapObjectId: true\n },\n\n aggregation: {\n allowDiskUse: true,\n batchSize: 200\n },\n};\n\n```\n\n**⚠️ All parameters are required.**\n\n4. Run from the project root:\n \n```bash\nnpm run mongoCollector\n\n```\n\n### Example of work\n\nSource collection users (from source):\n\n```js\n{ \"_id\": ObjectId(\"68a8c8207090be6dd0e23a90\"), \"name\": \"Alice\" }\n{ \"_id\": ObjectId(\"68a8c8207090be6dd0e23a91\"), \"name\": \"Sarah\" }\n{ \"_id\": ObjectId(\"68a8c8207090be6dd0e23a92\"), \"name\": \"John\" }\n\n```\n\nAfter running mongo-collector, in the target collection usersIdFromCrystal:\n\n```js\n{ \"users\": [ \"68a8c8207090be6dd0e23a90\", \"68a8c8207090be6dd0e23a91\", \"68a8c8207090be6dd0e23a92\" ] }\n```\n### Config parameters\n\n#### match\n\nYou can do any match configurations, for example:\n\nmatch: {} - take all documents.\n\nmatch: { createdAt: { $gte: new Date(\"2025-08-20T01:26:11.327+00:00\") } } - filter documents by date.\n\n#### documentId\n\ndocumentId: false - create a new document.\n\ndocumentId: '68a8c8207090be6dd0e23a90' - append data to an existing document, or create one with this _id if missing.\n\n#### rewriteDocuments\n\nrewriteDocuments: true - clear the entire target collection before writing.\n\n#### rewriteArray\n\ntrue - overwrite array \nfalse - append to an existing array\n\n#### duplicatesInArray\n\nfalse - eliminate duplicates (uses $addToSet) \n\n#### unwrapObjectId\n\ntrue - ObjectId('68a8c8207090be6dd0e23a90') → '68a8c8207090be6dd0e23a90' (final result in target).\n\n#### allowDiskUse\n\ntrue - allows MongoDB to write temporary data to disk when processing aggregation stages.false - restricts processing to memory only.batchSize: 10 - controls the length of the array inside each target document.\n\n**⚠️ Make sure the array does not exceed 16MB, otherwise MongoDB will throw an error.** \n\n \n\n