{"_id":"@bmacnaughton/simple-bench","_rev":"2-23f747c2c5f6f6dc50db37b82be88e96","name":"@bmacnaughton/simple-bench","dist-tags":{"latest":"2.1.1"},"versions":{"2.0.0":{"name":"@bmacnaughton/simple-bench","version":"2.0.0","description":"benchmarking runner for node.js","main":"index.js","scripts":{"start":"node index.js","test":"mocha '**/*.test.js'"},"keywords":["benchmark","node.js","benchmarking","benchmarking runner","tool"],"author":{"name":"Bruce A. MacNaughton"},"license":"MIT","devDependencies":{"chai":"^4.3.7","eslint":"^8.40.0","mocha":"^10.2.0"},"gitHead":"0f70a35c268f50fff5661029f719da3d00be28e6","_id":"@bmacnaughton/simple-bench@2.0.0","_nodeVersion":"16.19.1","_npmVersion":"8.19.3","dist":{"integrity":"sha512-jnHT6er9TrOSeDq6nInGW6C0fWD2ldVyABENUhK1UO8c5wDGDyhti+NNlJqvEQZs4IgaHKra0MkQCUc+prnLqw==","shasum":"d6008c93d76f9d71c726940acd29f50a4f6df723","tarball":"https://registry.npmjs.org/@bmacnaughton/simple-bench/-/simple-bench-2.0.0.tgz","fileCount":7,"unpackedSize":57916,"signatures":[{"keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA","sig":"MEQCIHn+gkhd4XdBaLc9liXWCj6DIvoErlI6q1BVIq1RwVYWAiAvwSbH6IxdHpOJzGQ/lWoOGogp/Do+XqZKXl2XVq8kvQ=="}]},"_npmUser":{"name":"bmacnaughton","email":"bmacnaughton@gmail.com"},"directories":{},"maintainers":[{"name":"bmacnaughton","email":"bmacnaughton@gmail.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages","tmp":"tmp/simple-bench_2.0.0_1683588915014_0.874930409333434"},"_hasShrinkwrap":false},"2.1.0":{"name":"@bmacnaughton/simple-bench","version":"2.1.0","description":"benchmarking runner for node.js","repository":{"type":"git","url":"git+https://github.com/bmacnaughton/simple-bench.git"},"main":"index.js","scripts":{"start":"node index.js","test":"mocha '**/*.test.js'"},"keywords":["benchmark","node.js","benchmarking","benchmarking runner","tool"],"author":{"name":"Bruce A. MacNaughton"},"license":"MIT","devDependencies":{"chai":"^4.3.7","eslint":"^8.40.0","mocha":"^10.2.0"},"gitHead":"acb5a487e074581d7292b0140a0710031adbc483","bugs":{"url":"https://github.com/bmacnaughton/simple-bench/issues"},"homepage":"https://github.com/bmacnaughton/simple-bench#readme","_id":"@bmacnaughton/simple-bench@2.1.0","_nodeVersion":"20.2.0","_npmVersion":"9.6.6","dist":{"integrity":"sha512-PMbucTC/HZT0R2P0PnJ5Wbd+oc11fMf07JGoi4kMFADXcwMgA8qdV3qrAHBr44nnGY8NwkQ7KDkp61rKVZgpvA==","shasum":"07a897b6397533f0627d86ee0f0b4d950f84cb7b","tarball":"https://registry.npmjs.org/@bmacnaughton/simple-bench/-/simple-bench-2.1.0.tgz","fileCount":7,"unpackedSize":53948,"signatures":[{"keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA","sig":"MEUCIEYwU9XqNtBBDkqgXtKkQmFTNUTjL8OuASc6sijX9ug8AiEA62qvvPWUOQSzYOebEkMyZbLAd+eIVdPZzBZzHJ9h1F4="}]},"_npmUser":{"name":"bmacnaughton","email":"bmacnaughton@gmail.com"},"directories":{},"maintainers":[{"name":"bmacnaughton","email":"bmacnaughton@gmail.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages","tmp":"tmp/simple-bench_2.1.0_1685837174805_0.9379741996406141"},"_hasShrinkwrap":false},"2.1.1":{"name":"@bmacnaughton/simple-bench","version":"2.1.1","description":"benchmarking runner for node.js","repository":{"type":"git","url":"git+https://github.com/bmacnaughton/simple-bench.git"},"main":"index.js","scripts":{"start":"node index.js","test":"mocha '**/*.test.js'"},"keywords":["benchmark","node.js","benchmarking","benchmarking runner","tool"],"author":{"name":"Bruce A. MacNaughton"},"license":"MIT","devDependencies":{"chai":"^4.3.7","eslint":"^8.40.0","mocha":"^10.2.0"},"gitHead":"f43ee560660f11f0450f8f8264b51e7966f58bfb","bugs":{"url":"https://github.com/bmacnaughton/simple-bench/issues"},"homepage":"https://github.com/bmacnaughton/simple-bench#readme","_id":"@bmacnaughton/simple-bench@2.1.1","_nodeVersion":"20.2.0","_npmVersion":"9.6.6","dist":{"integrity":"sha512-8Sv/NOk2W6EXUjx7cUFD4FzVVZxxRwBtv9XdIUvii4ev55Cv44Wqe639cWqDaVhgaGZyK7bwp7JpcMYWrqrf1Q==","shasum":"4cea534327e84004b97cecb83877c718bb9f92b3","tarball":"https://registry.npmjs.org/@bmacnaughton/simple-bench/-/simple-bench-2.1.1.tgz","fileCount":9,"unpackedSize":62658,"signatures":[{"keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA","sig":"MEYCIQD0N21ufzK09zjvR8xsT9qrVRVarmDWt1zU4huIigmvnAIhAILOvrR32Hn1WEd/iSo6FuQjueBAKbkxLlQ6WebsZbIE"}]},"_npmUser":{"name":"bmacnaughton","email":"bmacnaughton@gmail.com"},"directories":{},"maintainers":[{"name":"bmacnaughton","email":"bmacnaughton@gmail.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages","tmp":"tmp/simple-bench_2.1.1_1685837638021_0.13193897722618453"},"_hasShrinkwrap":false}},"time":{"created":"2023-05-08T23:35:14.942Z","2.0.0":"2023-05-08T23:35:15.169Z","modified":"2023-06-04T00:13:58.283Z","2.1.0":"2023-06-04T00:06:15.001Z","2.1.1":"2023-06-04T00:13:58.182Z"},"maintainers":[{"name":"bmacnaughton","email":"bmacnaughton@gmail.com"}],"description":"benchmarking runner for node.js","keywords":["benchmark","node.js","benchmarking","benchmarking runner","tool"],"author":{"name":"Bruce A. MacNaughton"},"license":"MIT","readme":"# simple-bench\n\nsimple-bench is a toolkit for executing JavaScript benchmarks; it is not a\nproduct that makes everything super-simple. It's been my experience that\nbenchmarking requires thought and is rarely super-simple. So simple-bench\ntries to handle the basics and make it relatively easy to manipulate common\nsettings.\n\nsimple-bench handles basic benchmarking tasks\n- warmup\n- multiple runs\n- removing statistical outliers\n- collecting timing and garbage-collection data\n- reporting as text or JSON\n\nWhile it is an npm package, it's not particularly convenient to install it\nas a dependency. Keep reading.\n\n## how to use - a recipe\n\n- create a directory for building your benchmark (or make a subdirectory in an existing\nproject).\n- fetch the simple-bench package from npm using `npm pack @bmacnaughton/simple-bench` and\nextract it where you want to work with it.\n- using `benchmarks/definitions.js` as an example, create your own benchmarks/definitions.js\nfile. it should export an object with the key `tests`. `tests` value is an object where\neach key is a function to be benchmarked.\n- run `node index.js key` to run a benchmark function. `key` is one of the keys\nin the `tests` object that was exported.\n- function-chains: if the test functions are structured to take an argument then\nmultiple functions can be given on the command line and their executions are\nchained, e.g., `node index.js func1 func2`. in this example `func2` will be passed\nthe return value of `func1`. this enables trying many different combinations without\nhaving to hardcode a function for each.\n- the function chain allows async functions; you can also just return a promise but it\nwill get an async wrapper (async function detection uses the constructor name).\n- the function `noop` is predefined and can be used to evaluate the cost of a function\nin a function-chain or of the framework itself.\n- if the work done by the function is very small relative to the work done by simple-bench,\nthen it may be necessary to loop within the test function to offset simple-bench's base\nlevel overhead.\n\n## options\n\nCommand line (ENV) options:\n- -d, --debug (DEBUG) - additional details written to stdout\n- --json (JSON) - output JSON, not text\n- -m, --memory (MEMORY) - capture memory usage stats too (not typically useful)\n- -t --terse (TERSE) - less output in text mode\n- --verbose (VERBOSE) - output some extra stuff\n- --verify (VERIFY) - set all iteration counts to 1 and wait MS to 10.\n\nThe previous options are all boolean; if the env var exists, the option is set, even if the\nvalue is empty.\n\nEnvironment-variable-only option:\n- BENCH - use this value for benchmark file (default is benchmark/definitions.js).\n\n## JSON output\n\nTwo lines per benchmark are output. The first is a short summary of what is being run;\nits primary purpose is visual feedback. The second is the benchmark results; it includes\nall the information from the summary as well as the following:\n\n- garbage collection stats\n- group stats\n  - the time group's execution\n  - the mean of the group run times\n  - the standard deviation of the group run times\n  - and the mean per iteration (mean divided by group iterations)\n- outliers (groups falling more than the specified standard deviation range from\nthe mean. this can be set in the benchmark definitions config.)\n  - each outlier group's time\n- clean (only groups falling within the specified standard deviation range)\n  - the times for each group's execution\n  - the mean of the group run times\n  - the standard deviation of the group run times\n  - the low cutoff value\n  - the high cutoff value\n\nExample (formatted; output is not):\n```json\n{\n  \"params\": {\n    \"functionChain\": [\"tinyText\", \"split\"],\n    \"warmupIterations\": 10,\n    \"groupIterations\": 100,\n    \"groupCount\": 5,\n    \"groupWaitMS\": 100,\n    \"stddevRange\": 2\n  },\n  \"gc\": {\n    \"count\": 2,\n    \"time\": 2.8927321434020996\n  },\n  \"raw\": {\n    \"times\": [\n      1.8154609203338623,\n      2.7900259494781494,\n      0.5000760555267334,\n      0.9043412208557129,\n      0.5999069213867188\n    ],\n    \"mean\": 1.3219622135162354,\n    \"meanPerIteration\": 0.013219622135162354,\n    \"stddev\": 0.8683340488055122\n  },\n  \"clean\": {\n    \"times\": [\n      1.8154609203338623,\n      2.7900259494781494,\n      0.5000760555267334,\n      0.9043412208557129,\n      0.5999069213867188\n    ],\n    \"lowRange\": 0,\n    \"highRange\": 3.05863031112726,\n    \"mean\": 1.3219622135162354,\n    \"meanPerIteration\": 0.013219622135162354,\n    \"stddev\": 0.8683340488055122\n  },\n  \"outliers\": {\n    \"times\": []\n  }\n}\n```\n\n## text output\n\nText output is intended for reading.\n\nExample:\n```bash\n[function chain: tinyText, split]\n[100 iterations x 5 groups (100ms intergroup pause)]\n[gc count: 2, gc time: 4.031]\n[group times: 1.87, 3.97, 0.87, 0.87, 0.56]\n[raw group mean 1.629 stddev 1.252 (0.016 per iteration)]\n[all group times within 0.00 to 4.13 (1.629 +/- 2 * 1.252)]\n[mean: 0.01629 per iteration]\n```\n\nIf there are outliers, the output is a little different:\n```bash\n[function chain: bigText, split]\n[100 iterations x 5 groups (100ms intergroup pause)]\n[gc count: 22, gc time: 7.594]\n[group times: 6.17, 5.26, 4.48, 3.74, 3.83]\n[raw group mean 4.699 stddev 0.917 (0.047 per iteration)]\n[excluding times outside 4.699 +/- 0.92: 6.17, 3.74]\n  [clean group mean 4.526 (0.045 per iteration) stddev 0.585]\n[mean: 0.04526 per iteration]\n```\n\n## benchmark/definitions.js in more detail\n\nSee the example benchmark in `benchmarks/definitions.js`. The goal of the benchmark\nis to compare different ways of splitting a log file into individual lines.\n\nKey points:\n- data initialization is done as part of module loading. This minimizes any impact\non the benchmark.\n- there are four sources, a giant file and a tiny file, each available as a string\nand as a buffer.\n- there are three consumers for the string representation and one for the buffer\nrepresentation.\n- `simple-bench.sh` is hand-coded to run each desired combination of source and\nconsumer.\n\nTypical output of `simple-bench.sh` is:\n\n```bash\n$ ./simple-bench.sh definitions\n[function chain: tinyText, split]\n[100000 iterations x 10 groups (1000ms intergroup pause)]\n[gc count: 986, gc time: 77.888]\n[group clean mean 85.779 raw stddev 5.479]\n[mean: 0.0008578 per iteration]\n\n[function chain: tinyText, regex]\n[100000 iterations x 10 groups (1000ms intergroup pause)]\n[gc count: 1061, gc time: 83.476]\n[group clean mean 148.547 raw stddev 3.462]\n[mean: 0.001485 per iteration]\n\n[function chain: tinyText, lastIxString]\n[100000 iterations x 10 groups (1000ms intergroup pause)]\n[gc count: 969, gc time: 74.153]\n[group clean mean 60.552 raw stddev 3.701]\n[mean: 0.0006055 per iteration]\n\n\n[function chain: bigText, split]\n[100000 iterations x 10 groups (1000ms intergroup pause)]\n[gc count: 5859, gc time: 444.208]\n[group clean mean 607.719 raw stddev 30.326]\n[mean: 0.006077 per iteration]\n\n[function chain: bigText, regex]\n[100000 iterations x 10 groups (1000ms intergroup pause)]\n[gc count: 3426, gc time: 511.623]\n[group clean mean 2281.062 raw stddev 60.194]\n[mean: 0.02281 per iteration]\n\n[function chain: bigText, lastIxString]\n[100000 iterations x 10 groups (1000ms intergroup pause)]\n[gc count: 5342, gc time: 399.592]\n[group clean mean 478.940 raw stddev 28.984]\n[mean: 0.004789 per iteration]\n\n\n[function chain: tinyBuffer, lastIxBuffer]\n[100000 iterations x 10 groups (1000ms intergroup pause)]\n[gc count: 986, gc time: 73.766]\n[group clean mean 118.157 raw stddev 3.669]\n[mean: 0.001182 per iteration]\n\n\n[function chain: bigBuffer, lastIxBuffer]\n[100000 iterations x 10 groups (1000ms intergroup pause)]\n[gc count: 5034, gc time: 479.090]\n[group clean mean 1932.526 raw stddev 10.916]\n[mean: 0.01933 per iteration]\n```\n\nFrom the output, it can be seen that the `lastIxString` approach is fastest.\n\n## more detail (and the code)\n\nExecution times reported for each group exclude everything other than executing\nthe function-chain. The code:\n\n```js\nasync function test() {\n  // call the tester's setup\n  if (definitions.setup) {\n    await (async() => definitions.setup(config))();\n  }\n  if (groupSetup) {\n    await (async() => groupSetup(config))();\n  }\n  // warmup\n  for (let i = warmupIterations; i > 0; i--) {\n    await execute(functionChain);\n  }\n  await pause(groupWaitMS);\n\n  // execute x groups of y iterations with a pause after each group\n  for (let i = 0; i < groupCount; i++) {\n    // setup for the group\n    if (groupSetup) {\n      await (async() => groupSetup(config))();\n    }\n    perf.mark('start-iteration');\n    for (let i = groupIterations; i > 0; i--) {\n      await execute(functionChain);\n    }\n    perf.measure('iteration-time', 'start-iteration');\n    if (memory) {\n      mem[i] = process.memoryUsage();\n    }\n    await pause(groupWaitMS);\n  }\n\n  // final pause\n  await pause(groupWaitMS * 10);\n  return pause(groupWaitMS);\n}\n\n//\n// execute functionChains\n//\nasync function execute(fc) {\n  let lastResult = undefined;\n  for (let i = 0; i < fc.length; i++) {\n    lastResult = await fc[i](lastResult);\n  }\n}\n```\n\nThe garbage collection counts and times include everything after the requires and\nprogram initialization. it's not clear (to me anyway) how to identify which garbage\ncollections are associated with the code being benchmarked and which are not. the\nbest way to get a handle on the baseline garbage collections is to use the `noop`\nbuilt-in function as a baseline for comparison.\n\n## todo\n\n- update make-csv, pipe-extract, and pipe-make-csv\n- flesh out testing: verify stats, different stddev ranges, ...\n- release scripts\n- add timing for each component of function-chain\n- add time/sample-based observations (ala criterion)\n- add timestamp and definitions file name to JSON output\n- add total elapsed time to output\n- add user tag facility\n","readmeFilename":"README.md","homepage":"https://github.com/bmacnaughton/simple-bench#readme","repository":{"type":"git","url":"git+https://github.com/bmacnaughton/simple-bench.git"},"bugs":{"url":"https://github.com/bmacnaughton/simple-bench/issues"}}